A radar and vision fusion device and radar and vision fusion method for automatic capture of a rotor unmanned aerial vehicle

Abstract

The application provides a radar and vision fusion device and method for automatic capture of a rotary-wing unmanned aerial vehicle, and belongs to the field of high-altitude control of unmanned aerial vehicles. In order to solve the problem that the existing unmanned aerial vehicle control mode generally has a short distance and cannot capture a high-speed target unmanned aerial vehicle. The approximate direction information of the target unmanned aerial vehicle is obtained through a radio frequency detection sensor to guide the carrier to approach the target, when the distance enters the range of 50 meters, the wide-angle camera sensor and the detection radar start to work, the target and its position information are obtained through the detection radar, the image picture obtained by the wide-angle camera sensor is identified by an edge computing terminal to identify the real unmanned aerial vehicle target, and the interference such as a bug bird is eliminated, the target direction after coordinate transformation is shot by the unmanned aerial vehicle capture net shooting device controlled by a two-dimensional holder, and finally the purpose of unmanned aerial vehicle control in a sensitive area is achieved, and the application prospect is very ideal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle high-altitude control, in particular to a radar and visual fusion device for automatic capture of rotary-wing unmanned aerial vehicles and a radar and visual fusion method. BACKGROUND

[0002] Unmanned aerial vehicles, referred to as unmanned aerial vehicles, rotary-wing unmanned aerial vehicles are widely used in aerial photography, disaster relief, aerial detection, agricultural irrigation, and various activities, and there is a particularly strong demand for control of rotary-wing unmanned aerial vehicles. However, there are many difficulties in achieving accurate detection of rotary-wing unmanned aerial vehicles. Rotary-wing unmanned aerial vehicles have the characteristics of low navigation height, slow navigation speed, and small aircraft cross-section target. Among them, the low flight target height will cause serious background clutter interference of surrounding buildings during radar detection; the slow navigation speed of such targets relative to traditional aircraft and the small detectable target require a radar with good performance.

[0003] Traditional unmanned aerial vehicle control methods generally consist of target detection and unmanned aerial vehicle countermeasures. In the target detection part, existing technologies such as the existing patent "Unmanned Aerial Vehicle Pan-Tilt Camera Target Recognition and Tracking Method and System Based on Artificial Intelligence" (Application No. CN202311189736.3) track unmanned aerial vehicles through cameras and pan-tilt mechanisms, but they cannot meet the requirements of identifying and tracking another high-speed moving target at high speed. Also, the existing patent "Intelligent Tracking and Shooting System for Long-Distance High-Speed Moving Targets" (Application No. CN202111156436.6) is designed for long-distance targets. In terms of unmanned aerial vehicle countermeasures, "signal interference" and "GPS deception" are commonly used, but these methods have the disadvantage of short countermeasures distance. SUMMARY

[0004] The technical problem to be solved by the present application is:

[0005] To solve the problem of existing unmanned aerial vehicle control methods that are generally short in distance and cannot capture high-speed target unmanned aerial vehicles.

[0006] The technical solution adopted by the present application to solve the above technical problems is:

[0007] The application provides a radar and vision fusion device for automatic capture of a rotor unmanned aerial vehicle, which comprises a crossing machine top support arranged above a crossing machine, an edge computing terminal and an unmanned aerial vehicle controller are arranged on the crossing machine top support, a crossing machine front end support is arranged on the crossing machine top support, a multi-sensor sensing component is arranged on the crossing machine front end support, the multi-sensor sensing component comprises a detection radar, a wide-angle camera sensor and a radio frequency detection sensor, three detection radars with the same parameters are distributed in a triangular shape and are fixed on the crossing machine front end support through a radar fixing base, the radio frequency detection sensor is arranged at the center of the triangularly distributed detection radars, and the wide-angle camera sensor is connected to the side end of the crossing machine front end support through a camera sensor connecting frame.

[0008] A two-dimensional holder is connected to the crossing machine through a crossing machine bottom support, a capture net launching device is arranged at the movement end of the two-dimensional holder, and an electronic gyroscope is arranged on the capture net launching device.

[0009] A radar and vision fusion method of a radar and vision fusion device for automatic capture of a rotor unmanned aerial vehicle, comprising the following steps:

[0010] In step S100, after an unlicensed target unmanned aerial vehicle appears, a radio frequency detection sensor detects a 2.4 GHz radio frequency signal communicated between the target unmanned aerial vehicle and a remote controller for controlling the flight of the target unmanned aerial vehicle, and obtains an airspace azimuth angle of the target unmanned aerial vehicle.

[0011] In step S200, the crossing machine flies to the airspace azimuth direction of the target unmanned aerial vehicle obtained in step S100, hovers around the target unmanned aerial vehicle, and identifies the target unmanned aerial vehicle by using a detection radar and a wide-angle camera sensor.

[0012] In step S300, an image picture obtained by the wide-angle camera sensor after distortion removal is identified by a neural network model deployed on an edge computing terminal through pre-training.

[0013] In step S400, direction information of the target unmanned aerial vehicle is obtained through three-dimensional positioning, and the obtained direction information is fused with direction information obtained by the detection radar through a series of coordinate transformations, so as to obtain position information of the target unmanned aerial vehicle.

[0014] In step S500, the orientation direction of the capture net launching device is controlled by using a two-dimensional holder, so that the capture net launching device faces the target during the flight of the crossing machine, so as to satisfy the accurate capture of the target unmanned aerial vehicle.

[0015] In step S600, when the detection radar detects that the distance between the target unmanned aerial vehicle and the crossing machine enters a controllable range, a capture net is automatically launched, so as to achieve the purpose of unmanned aerial vehicle control.

[0016] Further, in step S300, the following steps are included:

[0017] S310, calibrating the wide-angle camera sensor to obtain the internal and external parameters and distortion matrix of the wide-angle camera sensor; using a black and white chessboard calibration method, the Harris algorithm is used to detect the corner points of black and white changes, wherein the detection function of the corner points is simplified as:

[0018]

[0019] wherein, is a window function; is an energy function; is to reflect the movement of the window; is a pixel point coordinate; is a gray value function; is a real symmetric matrix;

[0020] S320, according to the internal and external parameters of the wide-angle camera sensor obtained in step S310, performing coordinate transformation operation on the panoramic picture collected by the wide-angle camera sensor, decomposing into two steps of rotation transformation and translation transformation, setting the center of the wide-angle camera sensor as the origin, taking the vertical direction of the wide-angle camera sensor as the Z axis, establishing a left-hand system, and defining it as a camera coordinate system, then the relationship between the camera coordinate system and the actual space coordinate system is as follows:

[0021]

[0022] wherein, is the coordinate in the camera coordinate system, is the coordinate in the actual space, R matrix is a rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values of the rotation angle in each direction; T is the translation transformation amount, that is, the external parameter in step S310;

[0023] The linear coordinate is expressed as:

[0024]

[0025] Thus, the required camera coordinate system is formed;

[0026] S330, performing a de-distortion operation to establish a non-linear distortion model:

[0027]

[0028] wherein, is the internal parameter obtained in step S310; is a distortion parameter; is the coordinate after de-distortion; is the Euclidean distance between two points;

[0029] The eccentric distortion is:

[0030]

[0031] The radial distortion is:

[0032]

[0033] According to the actual dedistortion effect, the distortion model parameters are continuously optimized.

[0034] Furthermore, if the visibility is low, the echo data of the detection radar is used to perform short-time Fourier transform (STFT) to obtain micro-Doppler information, and compared with the stored micro-Doppler data of the UAV and interference target, and then accurately identified as a UAV target.

[0035] Furthermore, in step S400, it includes:

[0036] S410: The wide-angle camera sensor transmits the RGB color information of the video image in units of video frames to the edge computing terminal, and the edge computing terminal identifies a series of related frame signals and line signals of SoF and EoL.

[0037] S420, the video image obtained in step S410 is Frame Set to , then the coordinates of the target UAV in the camera coordinate system are defined as , the target detection model deployed on the edge computing terminal obtains the pixel coordinates of the target drone as , coordinate transformation between two coordinate systems;

[0038] S430: Get the coordinates of the target drone in the camera coordinate system in step S420 Then, use the coordinates to calculate the horizontal rotation angle 𝜃 and vertical rotation angle 𝜙 of the target UAV to obtain the direction information of the target UAV in the actual space;

[0039] S440: The detection radar communicates with the edge computing terminal and simultaneously obtains information on the target distance, horizontal offset angle, and vertical offset angle;

[0040] In step S450, the edge computing terminal matches the collected radar sensor information with the coordinate information obtained by the method in step S420 using the image returned by the camera sensor, and uses the micro-Doppler spectrum obtained by short-time Fourier transform (STFT) of the echo data of the detection radar to filter out invalid information.

[0041] Further, in step S440, before the signal is returned to the target of the detection radar, a constant false alarm probability detection is used to detect the optimal detection threshold For:

[0042]

[0043] wherein, is the clutter interference power;

[0044] Using the improved OS-CFAR algorithm, the reference cells are first sorted from small to large, and the kth sample cell after sorting is used as the background clutter interference,

[0045] The value of k is determined according to the following product:

[0046]

[0047] At this time, the false alarm probability is:

[0048]

[0049] wherein, r1 is a self-defined ratio, is the factorial; is the combination number; is the reference cell length; is the threshold factor; is the total number of sample cells.

[0050] Further, in step S450, when the data of the detection radar is matched with the image target recognition data returned by the wide-angle camera sensor, the RGB format video frame data returned by the wide-angle camera sensor is processed by the deep learning model deployed in the edge computing terminal to obtain target category information, coordinate position information in the camera coordinate system, target pixel area size information, and confidence of the target unmanned aerial vehicle and the interference target; the coordinate information is transformed through step S420 to make the target coordinates obtained by the wide-angle camera sensor and the target coordinates obtained by the detection radar in the same coordinate system;

[0051] For the radar information obtained by the detection radar, the radar target information is matched with the wide-angle camera sensor video frame, the radar data is processed by proportional interpolation, and then the collection time of the two sensors is synchronized and matched; then the two sensor data at the same time is obtained for spatial matching, and the obtained targets with a difference of less than 3 pixel points in the same coordinate system are merged; then a vector is used to represent the merged target, at this time, the vector contains radar and visual sensor data, and finally the purpose of radar and visual fusion is achieved.

[0052] Further, in the coordinate transformation of step S320, the picture calibration of the camera sensor, the radar vision fusion process of step S420, the two sensor data point clouds in the same coordinate system, and the conversion of the camera coordinates into real-world orientation information of steps S500 and S600 to guide the two-dimensional pan-tilt steering and the emission of the capture net, the coordinate transformation is performed using the following model:

[0053]

[0054] wherein, is the intrinsic matrix of the wide-angle camera sensor, , , , is the intrinsic coefficient;

[0055] Let be defined as , let be defined as , and formula (10) is simplified as:

[0056] .

[0057] Further, the antenna arrangement of the detection radar is a subarray, which is arranged in a rectangular manner with a half-wavelength λ / 2 interval. Assuming that the subarrays are A, B, C, and D from left to right and from top to bottom, the detection radar realizes DOA estimation of the target through multiple groups of receiving antennas, and calculates the phase difference to obtain the target angle information:

[0058]

[0059] wherein, is the target angle of arrival, is the signal wavelength, is the phase difference between the two antennas, is the antenna spacing;

[0060] Let the path difference of the radar signal be , is the horizontal direction, is the vertical direction, let be the path and of the radar signal, and S be the received signal function, then the following relationship is obtained:

[0061]

[0062] wherein, is the horizontal radar signal path difference; is the vertical radar signal path difference;

[0063] The angular error is calculated by the following formula:

[0064]

[0065] wherein, is the horizontal azimuth error; is the vertical pitch error;

[0066] The error is subtracted to improve the accuracy of the detection radar in measuring the target in two horizontal directions and the vertical direction, and the accurate target azimuth information is obtained through the radar return data;

[0067] When the detection radar is working, the signal contacts the target unmanned aerial vehicle body, a part of which is reflected, and at the same time the millimeter wave radar receiving antenna continuously receives the echo with time delay reflected by the target unmanned aerial vehicle. The echo is mixed with the original transmitted signal through the mixer to obtain the intermediate frequency signal which is used for digital signal processing;

[0068] The signal output by the signal synthesizer is:

[0069]

[0070] The echo signal received by the receiver is:

[0071]

[0072] wherein,

[0073]

[0074] wherein, is the echo delay; is the distance between the millimeter wave radar and the target unmanned aerial vehicle; is the Doppler velocity; is the distance-related time; is the Doppler velocity-related time; is the Chirp signal slope; is the Chirp signal frequency; is the speed of light;

[0075] The intermediate frequency signal obtained through the mixer is:

[0076]

[0077] The derivative of the intermediate frequency signal is:

[0078]

[0079] wherein is the distance-dependent frequency; is the Doppler velocity-dependent frequency.

[0080] Further, in the control process of the two-dimensional holder in step S500, the edge computing terminal outputs a PWM signal with different duty cycles to the specific commutation signal of the motor driving module, so that the brushless motor maintains a constant torque, and finally drives the motor of the crossing machine to maintain a specific angle;

[0081] The SPWM signal using a sine wave and the six-axis electronic gyroscope output real-time acceleration and angular velocity information through the electronic gyroscope; using PID algorithm makes the motor movement more smooth:

[0082]

[0083]

[0084] wherein, is the actual angle position calculated according to the data returned by the electronic gyroscope and the target angle position difference; is the voltage output to the motor; is the static error; , , is the parameter adjusted according to the actual required response speed;

[0085] Sampling with T as the sampling period, we get:

[0086]

[0087] wherein, is the input deviation value at the kth sampling time, k is the sampling serial number, (k=0,1,2,…);

[0088] The coordinates of the detected target unmanned aerial vehicle camera coordinate system , the horizontal azimuth angle and the pitch azimuth angle of the target unmanned aerial vehicle are related as:

[0089]

[0090] .

[0091] Compared with the prior art, the present application has the following advantages:

[0092] The present application adopts the mode of crossing machine carrying unmanned plane net, and the model deployed in the edge computing terminal is used for target recognition of the target unmanned plane, and the unmanned plane net is turned through the two-dimensional holder, so as to perform the control operation of the target unmanned plane.

[0093] The target recognition and control system of the present application comprises a multi-sensor fusion perception device, a two-dimensional holder, an unmanned plane net launching device and an edge computing terminal. The multi-sensor comprises a radio frequency detection sensor, a millimeter wave radar and a wide-angle camera. The radio frequency detection sensor detects the 2.4GHz radio frequency signal emitted by the target unmanned plane to obtain the preliminary position information of the target unmanned plane, guiding the crossing machine to approach the target. The millimeter wave radar sensor is used in multiple ways to send the position information of the objects within the effective distance around to the edge computing terminal. The wide-angle camera sensor also sends the image picture to the edge computing terminal. The terminal performs target recognition and fuses the information obtained by the two sensors to accurately obtain the position information of the target unmanned plane after eliminating the interference objects such as birds, so as to control it.

[0094] The radar and vision fusion device for automatic capture of rotary-wing unmanned planes proposed in the present application uses the crossing machine to capture the target unmanned plane at high speed, solving the problem of short distance and inability to capture at high speed in the current unmanned plane control method. At the same time, the "radio frequency detection + visible light vision + millimeter wave radar" three-in-one unmanned plane target detection method is used to realize efficient and accurate identification and capture of the target unmanned plane. BRIEF DESCRIPTION OF DRAWINGS

[0095] Figure 1 It is a perspective view of a radar and vision fusion device for automatic capture of rotary-wing unmanned planes in an embodiment of the present application;

[0096] Figure 2 It is a model diagram of the target recognition range of the multi-sensor in an embodiment of the present application;

[0097] Figure 3 It is a flow chart of the radar and vision fusion method of the radar and vision fusion device for automatic capture of rotary-wing unmanned planes in an embodiment of the present application;

[0098] Figure 4 It is a working flow chart of the fusion of the radar sensor and the vision sensor in an embodiment of the present application;

[0099] Figure 5 It is a working flow chart of the wide-angle camera sensor used in an embodiment of the present application;

[0100] Figure 6 It is a rectangular distribution diagram of the radar antenna subarray in an embodiment of the present application.

[0101] MARKED FOR THE DRAWINGS:

[0102] 4, edge computing terminal; 11, front end support of crossing machine; 12, top support of crossing machine; 13, bottom support of crossing machine; 14, unmanned aerial vehicle controller; 21, detection radar; 22, radar fixing seat; 23, wide-angle camera sensor; 24, camera sensor connecting frame; 25, radio frequency detection sensor; 31, holder base; 32, two-dimensional holder; 33, net shooting device; 34, electronic gyroscope. DETAILED DESCRIPTION

[0103] In the description of the present application, it should be explained that the terms such as "up", "down", "front", "back", "left", "right" and other words indicating the orientation of the words in each embodiment only for the purpose of simplifying the description of the position relationship based on the drawings of the specification, and do not represent the components and devices indicated by the specific orientation and the operation and method, structure defined in the specification must be operated, this kind of orientation term does not constitute the limitation of the present application.

[0104] In order to make the above-mentioned purpose, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the drawings.

[0105] Specific implementation scheme one: combined Figure 1 As shown in the figure, the present application provides a kind of for rotary wing unmanned aerial vehicle automatic capture's radar vision fusion device, the top support of crossing machine 12 is arranged in the upper of crossing machine, and edge computing terminal 4 and unmanned aerial vehicle controller 14 are equipped on the top support of crossing machine 12, the unmanned aerial vehicle controller 14 includes unmanned aerial vehicle flight control, remote control receiver and power supply, the power supply can be lithium battery;The front end support of crossing machine 11 is equipped on the top support of crossing machine 12, and multiple sensor sensing components are equipped on the front end support of crossing machine 11, the multiple sensor sensing components include detection radar 21, wide-angle camera sensor 23 and radio frequency detection sensor 25, three detection radars 21 with same parameters are distributed in triangle shape, and are fixed on the front end support of crossing machine 11 by radar fixing seat 22, radar fixing seat 22 is used to prevent the falling caused by the shaking of detection radar 21 or wide-angle camera sensor 23 in the process of high-speed movement of crossing machine, the radio frequency detection sensor 25 is arranged at the center of the detection radar 21 distributed in triangle shape and extends the upper edge of detection radar 21, and the side end of the front end support of crossing machine 11 is connected with wide-angle camera sensor 23 by camera sensor connecting frame 24;

[0106] The detection radar 21 is a millimeter wave radar, the maximum effective detection distance is 40m-50m, the effective detection angle is ±60°, the frequency modulation continuous wave (FWCW) system technology is adopted, the millimeter wave radar is a wide beam millimeter wave radar, the triangular arrangement shape can realize the target detection around, is used for obtaining the target distance and speed information, and the echo data can also be processed to obtain the micro-Doppler data of the target to determine that the detected target is an unmanned aerial vehicle.

[0107] The wide-angle camera sensor 23 is used for obtaining the 120° wide-angle through machine air picture, the signal output by the wide-angle camera sensor 23 is in RGB format;

[0108] The radio frequency detection sensor 25 is used for receiving the 2.4GHz radio frequency signal of the target unmanned aerial vehicle, and obtaining the rough direction information of the target unmanned aerial vehicle;

[0109] The bottom of the through machine is provided with a through machine bottom support 13, the two-dimensional holder 32 is connected to the holder bottom 31 through the holder bottom 31, the motion end of the two-dimensional holder 32 is provided with a net shooting device 33, and the electronic gyroscope 34 is arranged on the net shooting device 33; the two-dimensional holder 32 is used for controlling the shooting direction of the net shooting device 33, and the direct current brushless motor control is used instead of the rudder control in order to make the load of the two rotating shafts smaller;

[0110] The shooting controllable range of the net shooting device 33 is 5 meters, the horizontal direction is 0°-360°, and the vertical direction is 0°-75°, so that the target below the through machine plane can be captured;

[0111] The electronic gyroscope 34 includes three-axis accelerometers of x-axis, y-axis and z-axis and three-axis angular rate meters of x-axis, y-axis and z-axis, and three-dimensional angle information can be output through the built-in algorithm;

[0112] The edge computing terminal 4 is used for deploying the model to identify the target unmanned aerial vehicle.

[0113] Specific implementation scheme two: combined Figures 1 to 5 As shown in the figure, the present application provides a radar and vision fusion method of a radar and vision fusion device for automatic capture of a rotary wing unmanned aerial vehicle, which comprises the following steps:

[0114] S100, after the appearance of the target unmanned aerial vehicle without permission, the radio frequency detection sensor 25 detects the 2.4GHz radio frequency signal communicated between the target unmanned aerial vehicle and the remote controller for controlling the flight, because the radio frequency detection sensor 25 internally uses an array containing multiple antennas, the angle of arrival can be obtained by calculating the phase difference of the received signal, and then a more accurate azimuth angle of the airspace where the target unmanned aerial vehicle is located can be obtained;

[0115] S200, the crossing machine flies in the airspace direction of the target UAV obtained in step S100, and hovers around the target UAV to identify the target UAV by using the detection radar 21 and the wide-angle camera sensor 23;

[0116] S300, in combination with the image picture obtained by the de-distorted wide-angle camera sensor 23, the neural network model deployed in the edge computing terminal 4 is identified by the pre-trained neural network model to identify the UAV target, including:

[0117] S310, in order to realize accurate three-dimensional positioning in the subsequent, the wide-angle camera sensor 23 needs to be calibrated to obtain the internal and external parameters and distortion matrix of the wide-angle camera sensor 23;

[0118] The black and white chessboard calibration method is adopted, and the Harris algorithm is used to detect the black and white change of the corner point, wherein the detection function of the corner point can be simplified as:

[0119]

[0120] Wherein, is a window function, which is equivalent to the weight of the corresponding coordinate; is an energy function, is used to reflect the movement of the window; is the pixel point coordinate, and the pixel point coordinate is summed in the window range; is a gray value function; after simplifying the quadratic form in the formula, the is a real symmetric matrix;

[0121] S320, according to the internal and external parameters of the wide-angle camera sensor 23 obtained in step S310, the coordinate transformation operation is performed on the panoramic picture collected by the wide-angle camera sensor 23, which can be decomposed into rotation transformation and translation transformation. The center of the wide-angle camera sensor 23 is set as the origin, the Z axis is perpendicular to the wide-angle camera sensor 23 and points outward, a left-handed system is established, and the camera coordinate system is defined as the camera coordinate system. The relationship between the camera coordinate system and the actual space coordinate system is as follows:

[0122]

[0123] Wherein, is the coordinate in the camera coordinate system, is the coordinate in the actual space, R matrix is a rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values of the rotation angle in each direction; T is the translation transformation amount, that is, the external parameter in step S310;

[0124] In addition, the linear coordinate can be expressed as:

[0125]

[0126] Thus, the required camera coordinate system is formed;

[0127] S330, this step is a distortion removal operation. The actual camera picture is not a complete linear pinhole model, so a nonlinear distortion model needs to be established. The radial distortion and the eccentric distortion are mainly considered, and the nonlinear model is established as follows:

[0128]

[0129] wherein, is the intrinsic parameter obtained in step S310; is a distortion parameter; is the coordinate after distortion removal; is the Euclidean distance between two points;

[0130] Specifically, the eccentric distortion is:

[0131]

[0132] The radial distortion is:

[0133]

[0134] According to the actual distortion removal effect, the distortion model parameters are continuously optimized until the technical requirements are met.

[0135] If the visibility is low, such as in foggy weather, the camera image cannot effectively identify the target. At this time, the echo data of the detection radar 21 is used to obtain the micro-Doppler information by short-time Fourier transform (STFT), and compared with the stored micro-Doppler data of unmanned aerial vehicles, birds or other interference, and then accurately identified as an unmanned aerial vehicle target.

[0136] S400, the direction information of the target unmanned aerial vehicle is obtained by three-dimensional positioning. The obtained direction information is fused with the direction information obtained by the detection radar 21 through a series of coordinate transformations, thereby forming a "radio frequency detection + visible light vision + millimeter wave radar" three-in-one unmanned aerial vehicle target detection method.

[0137] In the fusion process of radar and vision sensors, the wide-angle camera sensor 23 obtains a picture containing a suspected unlicensed target unmanned aerial vehicle and transmits it to the edge computing terminal 4 in real time. The edge computing terminal 4 processes the video frame information. At this time, for the coordinates of the suspected target unmanned aerial vehicle in the camera coordinate system, the longitudinal coordinate For the target object depth needs to be estimated separately; this is because, only the target object depth information is obtained, it is converted into each direction angle information to match the signal of the detection radar 21, and then the real target is captured again. The specific steps are as follows:

[0138] S410, the wide-angle camera sensor 23 transmits the RGB color information of the video picture in video frames to the edge computing terminal 4 through the MIPI high-speed interface, and the edge computing terminal 4 identifies the SoF and EoL series of related frame signals and line signals and performs the next step processing;

[0139] S420, the first frame of the video picture obtained in step S410 is called , and the coordinates of the target unmanned aerial vehicle in the camera coordinate system are defined as , and the target detection model deployed on the edge computing terminal 4 can obtain the coordinates of the suspected target unmanned aerial vehicle in the pixel coordinate system , and the coordinate transformation between the two coordinate systems is performed;

[0140] S430, after obtaining the coordinates of the suspected target unmanned aerial vehicle in the camera coordinate system in step S420 , the horizontal rotation angle θ and the vertical rotation angle φ of the target unmanned aerial vehicle are calculated using the coordinates, and the direction information of the suspected target unmanned aerial vehicle in the actual space is obtained;

[0141] S440, the detection radar 21 detects objects within a range of three 120° sectors connected end to end and within a distance of 50 meters, forms a target detection around, and then the detection radar 21 is connected with the RS485 bus and communicates with the edge computing terminal 4, and the target distance, horizontal offset angle and vertical offset angle information are obtained;

[0142] S450, the edge computing terminal 4 matches the radar sensor information collected and the camera sensor image returned using the coordinate information obtained by the method in step S420, further uses the millimeter wave radar echo data to filter out invalid information such as bird targets, and finally relies on the capture net launching device 33 for target unmanned aerial vehicle control operation;

[0143] S500, the orientation direction of the capture net launching device 33 is controlled by the two-dimensional holder 32, and since the capture net launching device 33 is installed on a two-dimensional holder 32, it can rotate to any horizontal and pitch angle within the controllable range, and then the capture net launching device 33 can face the target to meet the accurate capture of the target unmanned aerial vehicle;

[0144] ​S600, when the detection radar 21 monitors the distance of the target UAV into the controllable range, the UAV capture net is automatically launched, and the target UAV in the detection area is launched to achieve the purpose of UAV control.

[0145] Other combinations and connection relationships of the embodiment are the same as those of the specific embodiment.

[0146] Specific embodiment three: different from specific embodiment two, in step S440, before the back transmission target sensor, the detection radar 21 detects the target UAV, if the fixed threshold value decision is used for echo, the actual complex air clutter environment cannot be met, so the constant false alarm probability (CFAR) detection is used to present the real UAV target and further control operation,

[0147] Wherein, the optimal detection threshold is:

[0148]

[0149] Wherein, is the clutter interference power;

[0150] Using the improved OS-CFAR algorithm, the reference unit is sorted from small to large first, and the kth sample unit after sorting is used as the background clutter interference,

[0151] The value of k should be according to the following product:

[0152]

[0153] At this time, the false alarm probability is:

[0154]

[0155] Wherein, r1 is a self-defined ratio, indicates factorial; is the combination number; is the reference unit length; is the threshold factor; is the total number of sample units. Through this constant false alarm probability algorithm, the anti-interference ability for invalid targets can be improved in the air multi-target scene.

[0156] Specific implementation scheme four: different from specific implementation scheme two, in step S450, when matching the data of the detection radar 21 with the image target recognition data returned by the wide-angle camera sensor 23, the video frame data in RGB format returned by the wide-angle camera sensor 23 is subjected to a target category information, a coordinate position information in a target camera coordinate system, a target pixel area size information and a confidence degree of a target unmanned aerial vehicle and a jamming target by a deep learning model deployed in the edge computing terminal 4, and the above deep learning model is an existing model, and the method for obtaining the above information is also an existing method; these information forms a one-dimensional vector, and each target is represented by a vector, and in order to match the radar sensor data, the coordinate information is subjected to coordinate transformation through step S420, so that the target coordinates obtained by the wide-angle camera sensor 23 and the target coordinates obtained by the detection radar 21 are in the same coordinate system;

[0157] The method adopts a layer-by-layer progressive way of time matching, space matching and feature fusion, the detection radar returns target data every 50 ms, that is, 20 groups of radar data per second, and the wide-angle camera sensor 23 returns 30 frames of picture information per second, therefore, first, the radar target information is matched with the video frame of the wide-angle camera sensor 23, 30 groups of radar sensor data are obtained by proportionally interpolating the radar data, and the collection time of the two sensors is matched synchronously; then, the two sensor data at the same time are further subjected to space matching, and the targets with a difference of less than 3 pixel points in the horizontal and vertical coordinates in the same coordinate system obtained previously are merged; then, the merged target is represented by a vector, at this time, the vector contains the data of the radar and the camera, and finally the purpose of radar-camera fusion is achieved.

[0158] Specific implementation scheme five: different from specific implementation scheme four, in the coordinate transformation of step S320 for picture calibration of the camera sensor and removal of lens distortion, the radar-camera fusion process of step S420 for making the data points of the two sensors in the same coordinate system, and the conversion of the camera coordinates into real world orientation information of steps S500 and S600 for guiding the two-dimensional gimbal steering and the emission of the capture net, the following model can be used for coordinate transformation:

[0159]

[0160] It should be noted that the units of the above coordinates are all pixels, but the calculation of the subsequent direction angle is not affected; at this time, is the intrinsic matrix of the wide-angle camera sensor, 、 、 、 is the intrinsic coefficient; the coordinates of the suspected target unmanned aerial vehicle are calculated in the camera coordinate system ; if the former is defined as , the latter is defined as , formula (10) can be simplified as:

[0161] .

[0162] Specific implementation six: unlike specific implementation five, the antenna arrangement of the millimeter wave radar sensor is a subarray, which is arranged in a rectangular manner with a half-wavelength λ / 2 interval; in combination with FIG. 2, it is assumed that the subarrays are A, B, C, and D from left to right and from top to bottom, and the detection radar 21 realizes DOA estimation of the target through multiple groups of receiving antennas. In actual use, the target angle information is obtained by calculating the phase difference, which follows the following formula: Figure 6

[0163]

[0164] wherein, is the target angle of arrival, is the signal wavelength, is the phase difference between the two antennas, is the antenna spacing;

[0165] Let the path difference of the radar signal be , , which represents the horizontal direction and represents the vertical direction, and let be the path and of the radar signal, and S represents the received signal function, then the following relationship exists:

[0166]

[0167] Then the angle error can be calculated by the following formula:

[0168]

[0169] wherein, is the horizontal azimuth angle error; is the vertical pitch angle error;

[0170] Subtract the error to improve the accuracy of the detection radar 21 in measuring the target in two horizontal directions and the vertical direction, and obtain accurate target position information through radar return data.

[0171] ​Specific implementation seven: different from the specific implementation six, the working process of the detection radar 21 is that the signal synthesizer transmits a high-frequency modulated signal through the radio frequency antenna, the signal contacts the target unmanned aerial vehicle body, a part of it is reflected, at the same time, the millimeter wave radar receiving antenna continuously receives the target unmanned aerial vehicle reflected echo with time delay, the echo is mixed with the original transmitted signal through the mixer to obtain the intermediate frequency signal, and then the digital signal processing is used;

[0172] The signal output by the signal synthesizer can be represented as:

[0173]

[0174] The echo signal received by the receiver can be represented as:

[0175]

[0176] In the formula,

[0177]

[0178] Wherein, is the echo delay; is the distance between the millimeter wave radar and the target unmanned aerial vehicle; is the Doppler velocity; is the distance-related time; is the Doppler velocity-related time; is the Chirp signal slope; is the Chirp signal frequency; represents the speed of light;

[0179] The intermediate frequency signal obtained through the mixer is:

[0180]

[0181] The derivative of the intermediate frequency signal is:

[0182]

[0183] Wherein is the distance-related frequency; is the Doppler velocity-related frequency.

[0184] Specific implementation scheme eight: different from specific implementation scheme seven, in the control process of the two-dimensional holder 32 in step S500, the edge computing terminal 4 outputs a PWM signal with a different duty cycle to the motor drive module to keep the brushless motor at a constant torque, and finally drives the motor of the crossing machine to keep a specific angle;

[0185] In order to reduce the noise of the brushless motor when rotating, this process uses the SPWM signal of the sine wave; at the same time, the six-axis electronic gyroscope 34 is used to output real-time acceleration and angular velocity information, which is connected to the controller through the IIC communication interface; in addition, the PID algorithm is used to make the motor movement more smooth, and the algorithm can be expressed as:

[0186]

[0187]

[0188] Wherein, is the actual angle position calculated according to the data returned by the electronic gyroscope 34 and the target angle position difference; is the voltage output to the motor; is the static error; , , is a parameter adjusted according to the actual required response speed;

[0189] On this basis, sampling is carried out with T as the sampling period, and the following is obtained:

[0190]

[0191] Wherein is the input deviation value at the kth sampling time, k is the sampling serial number, (k=0, 1, 2, …);

[0192] The digital algorithm evolved from it is more convenient for the controller to process, and the difference between the target angle and the actual angle is used as a factor for real-time feedback, so as to realize the control of the brushless motor of the two-dimensional holder 32, so that the two-dimensional holder 32 can quickly respond to the target angle direction of the controller, and the direct direction points to the horizontal and pitch position of the target unmanned aerial vehicle, and the overall real-time performance is improved;

[0193] In this process, the coordinates of the detected target unmanned aerial vehicle camera coordinate system and the horizontal and pitch angle of the target unmanned aerial vehicle are related as follows:

[0194]

[0195] .

[0196] Although the present application has been disclosed with reference to the above examples, the scope of the present application is not limited to the above examples. Those skilled in the art to which the present application pertains will be able to make various changes and modifications without departing from the spirit and scope of the present application, and such changes and modifications will fall within the scope of the present application.

Claims

1. A radar-visual fusion method for a radar-visual fusion device for automatic capture of a rotary-wing UAV, characterized in that: The following steps are involved: S100, after the unauthorized target drone appears, the radio frequency detection sensor (25) detects the 2.4 GHz radio frequency signal communicated between the target drone and the remote controller controlling its flight, and obtains the airspace azimuth angle where the target drone is located; S200, the drone flies toward the airspace direction of the target drone obtained in step S100, hovers near the target drone, and uses the detection radar (21) and the wide-angle camera sensor (23) to identify the target drone; S300, combining the image obtained by the wide-angle camera sensor (23) after dedistortion, and identifying the target drone through a pre-trained neural network model deployed on the edge computing terminal (4); S400, obtaining direction information of the target UAV through three-dimensional positioning, fusing the obtained direction information with the direction information obtained by the detection radar (21) through a series of coordinate transformations, and obtaining the position information of the target UAV; S500, using the two-dimensional gimbal (32) to control the orientation of the net-launching device (33), so that the net-launching device (33) faces the target during the flight of the drone, thereby achieving accurate capture of the target drone; In the process of controlling the two-dimensional gimbal (32), the edge computing terminal (4) outputs PWM signals with different duty cycles to the motor drive module to generate specific commutation signals, so that the brushless motor maintains a constant torque, and finally drives the motor of the drone to maintain a specific angle; Using a sinusoidal SPWM signal and a six-axis electronic gyroscope (34), the electronic gyroscope (34) outputs real-time acceleration and angular velocity information; using a PID algorithm to make the motor movement smoother: in, The actual angular position calculated based on the data sent back by the electronic gyroscope (34) Angle position with target The difference between is the voltage output to the motor; is the static error; 、 、 It is a parameter adjusted according to the actual required response speed; Taking T as the sampling period for sampling, we get: in, is the deviation value input at the kth sampling moment, where k is the sampling sequence number (k=0,1,2,…); Coordinates of the detected target in the drone camera coordinate system , and the relationship between the horizontal azimuth and pitch azimuth of the target UAV is: S600, when the detection radar (21) detects that the distance between the target UAV and the crossing aircraft enters the controllable range, the capture net is automatically launched to achieve the purpose of UAV control; The radar-visual fusion device includes a top bracket (12) of a cross-country drone arranged above the cross-country drone, an edge computing terminal (4) and a drone controller (14) are arranged on the top bracket (12) of the cross-country drone, a front bracket (11) of the cross-country drone is arranged on the top bracket (12), a multi-sensor sensing component is arranged on the front bracket (11), and the multi-sensor sensing component includes a detection radar (21), a wide-angle camera sensor (23) and a radio frequency detection sensor (25), three detection radars (21) with the same parameters are distributed in a triangular shape and fixed on the front bracket (11) of the cross-country drone through a radar fixing seat (22), the radio frequency detection sensor (25) is arranged at the center of the triangularly distributed detection radars (21), and the side end of the front bracket (11) of the cross-country drone is connected to the wide-angle camera sensor (23) through a camera sensor connecting frame (24); A two-dimensional platform (32) is connected to the bottom of the drone via a bottom bracket (13) of the drone. A net-launching device (33) is provided at the moving end of the two-dimensional platform (32). An electronic gyroscope (34) is provided on the net-launching device (33).

2. The method for radar-visual fusion of a radar-visual fusion device for automatic capture of a rotary-wing UAV according to claim 1, characterized in that: In step S300, it includes: S310, performing a calibration operation on the wide-angle camera sensor (23) to obtain the internal and external parameters and the distortion matrix of the wide-angle camera sensor (23); using a black and white chessboard calibration method, using the Harris algorithm to detect the corner points of the black and white changes, wherein the detection function of the corner points is simplified to: in, is the window function; is the energy function; To reflect the movement of the window; is the pixel coordinate; is the gray value function; is a real symmetric matrix; S320, based on the internal and external parameters of the wide-angle camera sensor (23) obtained in step S310, perform a coordinate transformation operation on the panoramic image captured by the wide-angle camera sensor (23), decomposing it into two steps: rotation transformation and translation transformation. The center of the wide-angle camera sensor (23) is set as the origin, and the vertical wide-angle camera sensor (23) is pointed outward as the Z axis. A left-handed system is established and defined as the camera coordinate system. The relationship between the camera coordinate system and the coordinate system of the actual space is as follows: in, is the coordinate in the camera coordinate system, is the coordinate in the real space, R matrix is ​​the rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values ​​of the rotation angle in each direction; T is the translation transformation, that is, the external parameter in step S310; The linear coordinates are expressed as: This forms the required camera coordinate system; S330: Perform a dedistortion operation to establish a nonlinear distortion model: in, is the internal reference obtained in step S310; is the distortion parameter; is the coordinate after dedistortion; is the Euclidean distance between two points; The eccentric distortion is: The radial distortion is: According to the actual dedistortion effect, the distortion model parameters are continuously optimized.

3. The method for radar-visual fusion of a radar-visual fusion device for automatic capture of a rotary-wing UAV according to claim 2, characterized in that: If the visibility is low, the echo data of the detection radar (21) is used to perform short-time Fourier transform (STFT) to obtain micro-Doppler information, and compared with the stored micro-Doppler data of the UAV and the interference target, and then accurately identified as the UAV target.

4. The method for radar-visual fusion of a radar-visual fusion device for automatic capture of a rotary-wing UAV according to claim 3, characterized in that: In step S400, it includes: S410, the wide-angle camera sensor (23) transmits the RGB color information of the video image in units of video frames to the edge computing terminal (4), and the edge computing terminal (4) identifies a series of related frame signals and line signals of SoF and EoL; S420, the video image obtained in step S410 is Frame Set to , then the coordinates of the target UAV in the camera coordinate system are defined as , the target detection model deployed on the edge computing terminal (4) obtains the pixel coordinates of the target drone as , coordinate transformation between two coordinate systems; S430: Get the coordinates of the target drone in the camera coordinate system in step S420 Then, use the coordinates to calculate the horizontal rotation angle 𝜃 and vertical rotation angle 𝜙 of the target UAV to obtain the direction information of the target UAV in the actual space; S440, the detection radar (21) communicates with the edge computing terminal (4), and simultaneously obtains information on the target distance, horizontal offset angle, and vertical offset angle; S450, the edge computing terminal (4) matches the collected radar sensor information with the coordinate information obtained by the method in step S420 using the image sent back by the camera sensor, and uses the echo data of the detection radar (21) to perform a short-time Fourier transform (STFT) on the micro-Doppler spectrum to filter out invalid information.

5. The method for radar-visual fusion of a radar-visual fusion device for automatic capture of a rotary-wing UAV according to claim 4, characterized in that: In step S440, before the signal is transmitted back to the detection radar (21), a constant false alarm probability detection is used, and the optimal detection threshold is for: in, is the clutter interference power; The improved OS-CFAR algorithm is used, which includes sorting the reference units from small to large and using the kth sample unit after sorting as background clutter interference. The value of k is determined by multiplying as follows: The false alarm probability at this time is: Among them, r1 is the custom ratio, is factorial; is the number of combinations; is the reference unit length; is the threshold factor; is the total number of sample units.

6. The method for radar-visual fusion of a radar-visual fusion device for automatic capture of a rotary-wing UAV according to claim 5, characterized in that: In step S450, when matching the data of the detection radar (21) with the image target recognition data returned by the wide-angle camera sensor (23), the RGB format video frame data returned by the wide-angle camera sensor (23) is used to obtain target category information of the target drone and the interference target, coordinate position information of the target in the camera coordinate system, pixel area size information occupied by the target, and confidence level through the deep learning model deployed in the edge computing terminal (4); the coordinate information is transformed through step S420 so that the target coordinates obtained by the wide-angle camera sensor (23) and the target coordinates obtained by the detection radar (21) are in the same coordinate system; For the radar information obtained by the detection radar (21), the radar target information is matched with the video frame of the wide-angle camera sensor (23), and the radar data is proportionally interpolated and the acquisition time of the two sensors is synchronized; then the two sensor data at the same time are obtained for spatial matching, and the targets whose horizontal and vertical coordinates in the same coordinate system differ by less than 3 pixels are merged; then a vector is used to represent the merged target. At this time, the vector contains the data of both the radar and visual sensors, and finally the purpose of radar and visual fusion is achieved.

7. The method for radar-visual fusion of a radar-visual fusion device for automatic capture of a rotary-wing UAV according to claim 6, characterized in that: The following model is used for coordinate transformation in step S320 to calibrate the camera sensor image, in step S420 to make the two sensor data point clouds in the same coordinate system during radar vision fusion, and in steps S500 and S600 to convert the camera coordinates into real-world orientation information to guide the two-dimensional gimbal (32) to turn and launch the net: in, is the intrinsic parameter matrix of the wide-angle camera sensor, 、 、 、 is the internal parameter coefficient; Will Defined as ,Will Defined as , simplifying formula (10) to: 。 8. The method for radar-visual fusion of a radar-visual fusion device for automatic capture of a rotary-wing UAV according to claim 1, characterized in that: The antennas of the detection radar (21) are arranged as sub-arrays, with half-wavelength λ / 2 as intervals, in a rectangular arrangement; assuming that the sub-arrays are A, B, C, and D from left to right and from top to bottom, the detection radar (21) realizes the DOA estimation of the target through multiple groups of receiving antennas, and calculates the phase difference to obtain the target angle information: in, is the target arrival angle, is the signal wavelength, is the phase difference between the two antennas, is the antenna spacing; Assume the path difference of the radar signal is , is the horizontal direction, For the vertical direction, set is the distance and length of the radar signal, and S is the received signal function, then we have the following relationship: in, is the radar signal path difference in the horizontal direction; is the vertical radar signal path difference; The angular error is calculated using the following formula: in, is the horizontal azimuth error; is the vertical pitch angle error; The error is subtracted to improve the accuracy of the detection radar (21) in measuring the target in two horizontal directions and vertical directions, and accurate target direction information is obtained through the radar return data; When the detection radar (21) is working, the signal contacts the target UAV fuselage and a part of it is reflected. At the same time, the millimeter wave radar receiving antenna continuously receives the echo with a time delay reflected by the target UAV. The echo is mixed with the original transmission signal through the mixer to obtain an intermediate frequency signal, which is then used for digital signal processing. The signal output by the signal synthesizer is: The echo signal received by the receiver is: Where, in, is the echo delay; is the distance between the millimeter-wave radar and the target UAV; is the Doppler velocity; is the time associated with the distance; is the time associated with the Doppler velocity; is the slope of the Chirp signal; is the Chirp signal frequency; is the speed of light; The intermediate frequency signal obtained by mixing the mixer is: Taking the derivative of the intermediate frequency signal, we get: in is the distance-related frequency; is the frequency associated with the Doppler velocity.

Images