Radiovision fusion device and method for automatic capture of unmanned rotorcraft

Abstract

The invention provides a thunder-vision fusion device and a thunder-vision fusion method for automatic capture of a rotor unmanned aerial vehicle, and belongs to the field of unmanned aerial vehicle high-altitude control. The invention aims to solve the problems that the distance is generally short and a high-speed target unmanned aerial vehicle cannot be captured in the existing unmanned aerial vehicle control mode. The general direction information of the target unmanned aerial vehicle is obtained through the radio frequency detection sensor, the carrier crossing machine is guided to approach the target, when the distance is within 50 meters, the wide-angle camera sensor and the detection radar start to work, and the target and the position information thereof are obtained through the detection radar; according to an image obtained by the wide-angle camera sensor, the edge computing terminal identifies a real unmanned aerial vehicle target and eliminates interference of insects, birds and the like, the two-dimensional holder is matched to control the unmanned aerial vehicle catching net launching device to launch a catching net in the target direction after coordinate transformation, the purpose of unmanned aerial vehicle control in a sensitive area is finally achieved, and the method has a very ideal application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of high-altitude control of unmanned aerial vehicles, and in particular to a radar-vision fusion device and a radar-vision fusion method for automatically capturing a rotary-wing unmanned aerial vehicle. Background Art

[0002] Unmanned aerial vehicles are referred to as drones, and rotary-wing drones are widely used in various activities such as aerial photography, disaster relief, aerial detection, and agricultural irrigation. However, for sensitive locations such as government departments, university experimental sites, and military bases, there is a particularly strong demand for the control of rotary-wing drones. However, there are many difficulties in achieving accurate detection of rotary-wing drones. The flight targets of rotary-wing drones have the characteristics of low altitude, slow speed, and small cross-sectional targets. Among them, the low altitude of the flight target will cause serious background clutter interference from surrounding buildings during radar detection; this results in such targets having a slow navigation speed and small detectable targets relative to traditional aircraft, which requires the detection radar to have good performance.

[0003] Traditional drone control methods are generally divided into two aspects: target detection and drone countermeasures. In the target detection part, the existing technology, such as the existing patent based on artificial intelligence drone gimbal camera target identification and tracking method and system (application number: CN202311189736.3), tracks drones by adding a camera to a gimbal, but it cannot meet the requirements of identifying and tracking another high-speed moving target under high-speed motion; there is also an existing patent for an intelligent tracking and shooting system for long-distance high-speed moving targets (application number: CN202111156436.6), which targets long-distance targets. In terms of drone countermeasures, "signal interference" and "GPS spoofing" are used as means, but such methods generally have the disadvantage of short countermeasure distances. Summary of the invention

[0004] The technical problems to be solved by the present invention are:

[0005] In order to solve the problem that existing drone control methods generally have short distances and cannot capture high-speed target drones.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] The present invention provides a radar-vision fusion device for automatic capture of a rotary-wing unmanned aerial vehicle, comprising a top bracket of a cross-country drone arranged above the cross-country drone, an edge computing terminal and a drone controller being arranged on the top bracket of the cross-country drone, a front bracket of the cross-country drone being arranged on the top bracket of the cross-country drone, a multi-sensor sensing component being arranged on the front bracket of the cross-country drone, the multi-sensor sensing component comprising a detection radar, a wide-angle camera sensor and a radio frequency detection sensor, three detection radars with the same parameters being distributed in a triangular shape and being fixed on the front bracket of the cross-country drone through a radar fixing seat, the radio frequency detection sensor being arranged at the center of the detection radars distributed in a triangular shape, and a side end of the front bracket of the cross-country drone being connected to a wide-angle camera sensor through a camera sensor connecting frame;

[0008] A two-dimensional pan-tilt is connected to the bottom of the drone via a bottom bracket of the drone. A net-catching launching device is arranged at the moving end of the two-dimensional pan-tilt, and an electronic gyroscope is arranged on the net-catching launching device.

[0009] A method for automatically capturing a radar-vision fusion device for a rotary-wing unmanned aerial vehicle comprises the following steps:

[0010] S100: After an unauthorized target drone appears, the radio frequency detection sensor detects the 2.4 GHz radio frequency signal communicated between the target drone and the remote controller controlling its flight, and obtains the airspace azimuth angle where the target drone is located;

[0011] S200, the drone flies in the airspace direction of the target drone obtained in step S100, hovers near the target drone, and uses detection radar and wide-angle camera sensors to identify the target drone;

[0012] S300, combined with the image obtained by the wide-angle camera sensor after dedistortion, identifies the target drone through the pre-trained neural network model deployed on the edge computing terminal;

[0013] S400, obtaining direction information of the target UAV through three-dimensional positioning, fusing the obtained direction information with the direction information obtained by the detection radar through a series of coordinate transformations, and obtaining position information of the target UAV;

[0014] S500, using the two-dimensional gimbal to control the direction of the net-capturing launching device, so that the net-capturing launching device faces the target during the flight of the drone, so as to accurately capture the target drone;

[0015] S600: When the detection radar detects that the distance between the target UAV and the Flying Drone is within the controllable range, the capture net is automatically launched to achieve the purpose of UAV control.

[0016] Furthermore, in step S300, it includes:

[0017] S310, calibrate the wide-angle camera sensor to obtain the internal and external parameters and distortion matrix of the wide-angle camera sensor; use the black and white chessboard calibration method and the Harris algorithm to detect the corner points of black and white changes, where the detection function of the corner points is simplified to:

[0018]

[0019] Among them, w(x,y) is the window function; E(u,v) is the energy function; u,v represent the movement of the window; x,y are the pixel coordinates; I is the gray value function; M is a real symmetric matrix;

[0020] S320, according to the internal and external parameters of the wide-angle camera sensor obtained in step S310, coordinate transformation operation is performed on the panoramic image collected by the wide-angle camera sensor, which is decomposed into two steps of rotation transformation and translation transformation. The center of the wide-angle camera sensor is set as the origin, and the vertical wide-angle camera sensor is pointed outward as the Z axis. A left-handed system is established and defined as the camera coordinate system. The relationship between the camera coordinate system and the coordinate system of the actual space is as follows:

[0021]

[0022] in, is the coordinate in the camera coordinate system, is the coordinate in the actual space, R matrix is ​​the rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values ​​of the rotation angles in each direction; T is the translation transformation, i.e., the external parameter in step S310;

[0023] The linear coordinates are expressed as:

[0024]

[0025] This forms the required camera coordinate system;

[0026] S330, performing a dedistortion operation and establishing a nonlinear distortion model:

[0027]

[0028] Among them, a x ,a y ,u0,v0 are the internal parameters obtained in step S310; k1,k2,p1,p2 are distortion parameters; (u1,v1) are the coordinates after dedistortion; r is the Euclidean distance between two points;

[0029] The eccentric distortion is:

[0030]

[0031] The radial distortion is:

[0032]

[0033] According to the actual de-distortion effect, the distortion model parameters are continuously optimized.

[0034] Furthermore, if the visibility is low, the echo data of the detection radar is used to perform short-time Fourier transform (STFT) to obtain micro-Doppler information, and compared with the stored micro-Doppler data of the UAV and interference target, so as to accurately identify it as a UAV target.

[0035] Furthermore, in step S400, it includes:

[0036] S410, the wide-angle camera sensor transmits the RGB color information of the video screen in units of video frames, and transmits the information collected by the camera to the edge computing terminal, and the edge computing terminal identifies a series of related frame signals and line signals of SoF and EoL;

[0037] S420, set the i-th frame of the video image obtained in step S410 to I i , then the coordinates of the target drone in the camera coordinate system are defined as [X i ,Y i ,Z i ] T , the target detection model deployed on the edge computing terminal obtains the pixel coordinates of the target drone as [u i ,v i ] T , coordinate transformation between two coordinate systems;

[0038] S430: In step S420, the coordinates of the target UAV in the camera coordinate system are obtained [X i ,Y i ,Z i ] T Then, the coordinates are used to calculate the horizontal rotation angle θ and the vertical rotation angle φ of the target UAV, and the direction information of the target UAV in the actual space is obtained;

[0039] S440, the detection radar communicates with the edge computing terminal, and simultaneously obtains information on the target distance, horizontal offset angle, and vertical offset angle;

[0040] S450, the edge computing terminal matches the collected radar sensor information with the coordinate information obtained by the method in step S420 using the image sent back by the camera sensor, and uses the micro-Doppler spectrum after short-time Fourier transform (STFT) of the echo data of the detection radar to filter out invalid information.

[0041] Furthermore, in step S440, before the signal is transmitted back to the detection radar, a constant false alarm probability detection is adopted, and the optimal detection threshold T is:

[0042] T=-μ0 lnP fa (7)

[0043] Among them, μ0 is the clutter interference power;

[0044] The improved OS-CFAR algorithm is used, including sorting the reference cells from small to large, and using the sorted kth sample cell as background clutter interference.

[0045] The value of k is determined by multiplying as follows:

[0046] k=r1·N (8)

[0047] At this time, the false alarm probability is:

[0048]

[0049] Among them, r1 is the custom ratio, Γ is the factorial, k() is the number of combinations, N is the reference unit length, α is the threshold factor, and n is the total number of sample units.

[0050] Further, in step S450, when matching the data of the detection radar with the image target recognition data returned by the wide-angle camera sensor, the video frame data in RGB format returned by the wide-angle camera sensor obtains the target category information of the target UAV and the interference target, the coordinate position information of the camera coordinate system where the target is located, the pixel area size information occupied by the target, and the confidence level through the deep learning model deployed in the edge computing terminal; the coordinate information is transformed through step S420 so that the target coordinates obtained by the wide-angle camera sensor and the target coordinates obtained by the detection radar are in the same coordinate system;

[0051] For the radar information obtained by the detection radar, the radar target information is matched with the video frame of the wide-angle camera sensor, and the radar data is proportionally interpolated to synchronize the acquisition time of the two sensors; then the two sensor data at the same time are obtained for spatial matching, and the targets whose horizontal and vertical coordinates in the same coordinate system differ by less than 3 pixels are merged; then a vector is used to represent the merged target. At this time, the vector contains both the radar and visual sensor data, and finally achieves the purpose of radar and visual fusion.

[0052] Furthermore, when performing the coordinate transformation of step S320 to calibrate the image of the camera sensor, making the two sensor data point clouds in the same coordinate system during the radar vision fusion process of step S420, and converting the camera coordinates into real-world orientation information to guide the two-dimensional gimbal to turn and launch the net in steps S500 and S600, the following model is used for coordinate transformation:

[0053]

[0054] Among them, K is the intrinsic parameter matrix of the wide-angle camera sensor, f u 、f ν , u0, v0 are internal parameter coefficients;

[0055] Will [u i ,v i ] T Defined as p i , and [X i ,Y i ,Z i ] T Defined as P i , simplifying formula (10) to:

[0056] P i =Z i K -1 p i (11).

[0057] Furthermore, the antennas of the detection radar are arranged as sub-arrays, with half-wavelength λ / 2 as intervals, in a rectangular arrangement; assuming that the sub-arrays are A, B, C, and D from left to right and from top to bottom, the detection radar estimates the DOA of the target through multiple groups of receiving antennas, and calculates the phase difference to obtain the target angle information:

[0058]

[0059] Where θ is the target arrival angle, λ is the signal wavelength, is the phase difference between the two antennas, and d is the antenna spacing;

[0060] Assume that the distance difference of the radar signal is Δ, θ is the horizontal direction, φ is the vertical direction, Σ is the sum of the distances of the radar signal, and S is the received signal function, then we have the following relationship:

[0061]

[0062] Among them, Δ θ is the radar signal path difference in the horizontal direction; Δ φ is the radar signal path difference in the vertical direction;

[0063] The angular error is calculated by the following formula:

[0064]

[0065] Among them, Δβ is the horizontal azimuth error; Δε is the vertical pitch angle error;

[0066] Subtract the error to improve the accuracy of the detection radar in measuring the target in two horizontal and vertical directions, and obtain accurate target direction information through the radar return data;

[0067] When the detection radar is working, the signal contacts the fuselage of the target UAV, and a part of it is reflected. At the same time, the millimeter wave radar receiving antenna continues to receive the delayed echo reflected by the target UAV. The echo is mixed with the original transmission signal through the mixer to obtain an intermediate frequency signal, which is then used for digital signal processing.

[0068] The signal output by the signal synthesizer is:

[0069]

[0070] The echo signal received by the receiver is:

[0071]

[0072] In the formula,

[0073]

[0074] Among them, τ is the echo delay; R0 is the distance between the millimeter wave radar and the target UAV; ν is the Doppler velocity; is the distance-related time; t m is the time associated with Doppler velocity; μ is the slope of the Chirp signal; f c is the Chirp signal frequency; c is the speed of light;

[0075] The intermediate frequency signal obtained by mixing the mixer is:

[0076]

[0077] Taking the derivative of the intermediate frequency signal, we get:

[0078]

[0079] where f i is the distance-related frequency; f d is the frequency associated with the Doppler velocity.

[0080] Furthermore, in the process of controlling the two-dimensional gimbal in step S500, the edge computing terminal outputs PWM signals with different duty cycles to the motor drive module with specific commutation signals, so that the brushless motor maintains a constant torque, and finally drives the motor of the FPV to maintain a specific angle;

[0081] Use a sinusoidal SPWM signal and a six-axis electronic gyroscope to output real-time acceleration and angular velocity information; use a PID algorithm to make the motor movement smoother:

[0082] e(t)=y(t)-r(t) (20)

[0083]

[0084] Wherein, e(t) is the difference between the actual angle position r(t) calculated based on the data sent back by the electronic gyroscope and the target angle position y(t); u(t) is the voltage output to the motor; u0 is the static error; K p , T1, T D It is a parameter adjusted according to the actual required response speed;

[0085] Taking T as the sampling period for sampling, we get:

[0086]

[0087] Among them, e k is the deviation value input at the kth sampling time, k is the sampling sequence number, (k=0,1,2,…);

[0088] The coordinates of the detected target drone camera coordinate system [X i ,Y i ,Z i ] T , and the relationship between the horizontal azimuth and pitch azimuth of the target UAV is:

[0089]

[0090] Compared with the prior art, the present invention has the following beneficial effects:

[0091] The present invention adopts a method of carrying a drone net on a flying machine, identifies the target drone through a model deployed in an edge computing terminal, and steers the drone net through a two-dimensional gimbal to perform control operations on the target drone.

[0092] The target identification and control system of the present invention includes a multi-sensor fusion perception device, a two-dimensional gimbal and a UAV net launching device, and an edge computing terminal. The multi-sensors include a radio frequency detection sensor, a millimeter wave radar, and a wide-angle camera. The radio frequency detection sensor detects the 2.4GHz radio frequency signal emitted by the target UAV, obtains the preliminary position information of the target UAV, and guides the FVR to approach the target. Multiple millimeter wave radar sensors are used in combination to send the position information of objects within the effective distance around to the edge computing terminal. The wide-angle camera sensor also sends the image to the edge computing terminal. The terminal performs target identification and fuses the information obtained by the two sensors. After eliminating interference objects such as insects and birds, the position information of the target UAV is accurately obtained for its control.

[0093] The radar-vision fusion device for automatic capture of rotary-wing UAVs proposed in the present invention uses a cross-country drone to capture the target UAV during high-speed operation, which solves the problems of short distance and inability to capture at high speed of current UAV control methods. At the same time, it uses a three-in-one UAV target detection method of "radio frequency detection + visible light vision + millimeter wave radar" to achieve efficient and accurate identification and capture of target UAVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is a stereoscopic diagram of a radar-vision fusion device for automatic capture of a rotary-wing UAV in an embodiment of the present invention;

[0095] Figure 2 A schematic diagram of a model of a multi-sensor target recognition range in an embodiment of the present invention;

[0096] Figure 3 It is a flow chart of a radar-vision fusion method of a radar-vision fusion device for automatic capture of a rotary-wing UAV in an embodiment of the present invention;

[0097] Figure 4 This is a workflow diagram of the fusion of the radar sensor and the visual sensor in an embodiment of the present invention;

[0098] Figure 5 A flowchart of the wide-angle camera sensor used in an embodiment of the present invention;

[0099] Figure 6 4 is a rectangular distribution diagram of the sub-array of the radar antenna in an embodiment of the present invention.

[0100] Description of reference numerals:

[0101] 4. Edge computing terminal; 11. Front bracket of the drone; 12. Top bracket of the drone; 13. Bottom bracket of the drone; 14. UAV controller; 21. Detection radar; 22. Radar fixing seat; 23. Wide-angle camera sensor; 24. Camera sensor connecting frame; 25. Radio frequency detection sensor; 31. Gimbal base; 32. Two-dimensional gimbal; 33. Net launching device; 34. Electronic gyroscope. DETAILED DESCRIPTION

[0102] In the description of the present invention, it should be noted that the terminology nouns in the various embodiments, such as "up", "down", "front", "back", "left", "right", etc., which indicate directions, are only for simplifying the description of the positional relationship based on the drawings in the specification, and do not mean that the referred elements and devices must be operated according to the specific directions and defined operations and methods and structures in the specification. Such directional nouns do not constitute a limitation on the present invention.

[0103] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0104] Specific implementation plan 1: Combine Figure 1 As shown, the present invention provides a radar-vision fusion device for automatic capture of a rotary-wing UAV, wherein the top bracket 12 of the cross-country drone is arranged above the cross-country drone, and the top bracket 12 of the cross-country drone is provided with an edge computing terminal 4 and a UAV controller 14, and the UAV controller 14 includes a UAV flight control, a remote control receiver and a power supply, and the power supply can be a lithium battery; the top bracket 12 of the cross-country drone is provided with a cross-country drone front bracket 11, and the cross-country drone front bracket 11 is provided with a multi-sensor sensing component, and the multi-sensor sensing component includes a detection radar 21, a wide-angle camera sensor 23 and a radio frequency detection sensor 25, and three detection radars 21 with the same parameters are distributed in a triangular shape and fixed on the front bracket 11 of the cross-country drone through a radar fixing seat 22, and the radar fixing seat 22 is used to prevent the detection radar 21 or the wide-angle camera sensor 23 from falling off due to shaking during the high-speed movement of the cross-country drone, and the radio frequency detection sensor 25 is arranged at the center of the triangularly distributed detection radar 21 and extends out of the upper edge of the detection radar 21, and the side end of the front bracket 11 of the cross-country drone is connected to the wide-angle camera sensor 23 through a camera sensor connecting frame 24;

[0105] The detection radar 21 is a millimeter wave radar with a maximum effective detection distance of 40m-50m and an effective detection angle of ±60°. It adopts frequency modulated continuous wave FWCW system technology. The millimeter wave radar is a wide beam millimeter wave radar. The triangular arrangement shape can realize the detection of targets around it, which is used to obtain the target distance and speed information. The echo data can also be processed to obtain the micro Doppler data of the target to determine that the detected target is a drone;

[0106] The wide-angle camera sensor 23 is used to obtain a 120° wide-angle aerial image of the drone, and the signal output by the wide-angle camera sensor 23 is in RGB format;

[0107] The radio frequency detection sensor 25 is used to receive the 2.4Ghz radio frequency signal of the target UAV and obtain rough direction information of the target UAV;

[0108] A bottom bracket 13 of the drone is provided below the drone, and the bottom bracket 13 of the drone is connected to a two-dimensional gimbal 32 via a gimbal base 31. A net-catching launch device 33 is provided at the moving end of the two-dimensional gimbal 32, and an electronic gyroscope 34 is provided on the net-catching launch device 33. The two-dimensional gimbal 32 is used to control the launching direction of the net-catching launch device 33. In order to reduce the load on the two rotating axes, a DC brushless motor control is used instead of a steering gear control.

[0109] The launch controllable range of the capture net launch device 33 is 5 meters, and its operable angle is 0°-360° horizontally and 0°-75° vertically, and it can capture targets below the plane of the flying machine;

[0110] The electronic gyroscope 34 includes a three-axis accelerometer of the x-axis, y-axis and z-axis and a three-axis angular velocity meter of the x-axis, y-axis and z-axis, and can output the obtained three-dimensional angle information through a built-in algorithm;

[0111] The edge computing terminal 4 is used to deploy a model to identify the target drone.

[0112] Specific implementation plan 2: Combination Figures 1 to 5 As shown, the present invention provides a radar-vision fusion method for a radar-vision fusion device for automatic capture of a rotary-wing unmanned aerial vehicle, comprising the following steps:

[0113] S100, after the unauthorized target drone appears, the RF detection sensor 25 detects the 2.4Ghz RF signal communicated between the target drone and the remote controller controlling its flight. This is because the RF detection sensor 25 uses an array containing multiple antennas. The angle of arrival can be calculated by calculating the phase difference of the received signal, and then a more accurate airspace azimuth angle of the target drone can be obtained;

[0114] S200, the drone flies in the airspace direction of the target drone obtained in step S100, hovers near the target drone, and uses the detection radar 21 and the wide-angle camera sensor 23 to identify the target drone;

[0115] S300, combining the image obtained by the wide-angle camera sensor 23 after dedistortion, and identifying the drone target through the pre-trained neural network model deployed on the edge computing terminal 4, including:

[0116] S310: In order to accurately realize three-dimensional positioning later, it is necessary to calibrate the wide-angle camera sensor 23 to obtain the internal and external parameters and distortion matrix of the wide-angle camera sensor 23;

[0117] The black and white chessboard calibration method is used, and the Harris algorithm is used to detect the corner points of black and white changes. The detection function of the corner points can be simplified as follows:

[0118]

[0119] Among them, w(x,y) is the window function, which is equivalent to the weight of the corresponding coordinates; E(u,v) is the energy function, u,v is used to reflect the movement of the window; x,y are the pixel coordinates, and the pixel coordinates are summed within the window range; I is the gray value function; after being simplified to a quadratic form, M in the formula is a real symmetric matrix;

[0120] S320, according to the internal and external parameters of the wide-angle camera sensor 23 obtained in step S310, coordinate transformation operation is performed on the panoramic image captured by the wide-angle camera sensor 23, and the transformation can be decomposed into two steps: rotation transformation and translation transformation. The center of the wide-angle camera sensor 23 is set as the origin, and the vertical wide-angle camera sensor 23 pointing outward is used as the Z axis to establish a left-handed system, which is defined as the camera coordinate system. The relationship between the camera coordinate system and the coordinate system of the actual space is as follows:

[0121]

[0122] in, is the coordinate in the camera coordinate system, is the coordinate in the actual space, the R matrix is ​​the rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values ​​of the rotation angles in each direction; T is the translation transformation, i.e., the external parameter in step S310;

[0123] In addition, the linear coordinates can be expressed as:

[0124]

[0125] This forms the required camera coordinate system;

[0126] S330: This step is a de-distortion operation. The actual camera image is not a completely linear aperture model, so a nonlinear distortion model needs to be established, focusing on radial distortion and eccentric distortion. The established nonlinear model is as follows:

[0127]

[0128] Among them, a x ,a y ,u0,v0 are the internal parameters obtained in step S310; k1,k2,p1,p2 are distortion parameters; (u1,v1) are the coordinates after dedistortion; r is the Euclidean distance between two points;

[0129] Specifically, the eccentric distortion is:

[0130]

[0131] The radial distortion is:

[0132]

[0133] According to the actual de-distortion effect, the distortion model parameters are continuously optimized until the technical requirements are met;

[0134] If the visibility is low due to foggy weather, the camera image cannot effectively identify the target. In this case, the echo data of the detection radar 21 is used to perform short-time Fourier transform (STFT) to obtain micro-Doppler information, and compared with the stored micro-Doppler data of drones, birds or other interferences, and then accurately identified as drone targets;

[0135] S400, obtaining direction information of the target UAV through three-dimensional positioning, and fusing the obtained direction information with the direction information obtained by the detection radar 21 through a series of coordinate transformations, thereby forming a UAV target detection method of "radio frequency detection + visible light vision + millimeter wave radar";

[0136] During the fusion process of radar and vision sensors, the wide-angle camera sensor 23 obtains the image of the suspected unauthorized target drone and transmits it to the edge computing terminal 4 in real time. The edge computing terminal 4 processes the video frame information. At this time, for the coordinates of the suspected target drone in the camera coordinate system, the vertical coordinate Z C The depth of the target object needs to be estimated separately. This is because only by obtaining the depth information of the target object and converting it into angle information in each direction can it be matched with the signal of the detection radar 21, and then the capture operation can be performed on the real target. The specific steps are as follows:

[0137] S410, the wide-angle camera sensor 23 transmits the RGB color information of the video screen in units of video frames to the edge computing terminal 4 through the MIPI high-speed interface, and the edge computing terminal 4 identifies a series of related frame signals and line signals of SoF and EoL and performs the next step of processing;

[0138] S420, the i-th frame of the video image obtained in step S410 is called I i , then the coordinates of the target drone in the camera coordinate system are defined as [X i ,Y i ,Z i ] T , the target detection model deployed on the edge computing terminal 4 can obtain the coordinates of the suspected target drone in the pixel coordinate system as [u i ,v i ] T , coordinate transformation between two coordinate systems;

[0139] S430: In step S420, the coordinates of the suspected target drone in the camera coordinate system are obtained [X i ,Y i ,Z i ] T Then, the coordinates are used to calculate the horizontal rotation angle θ and the vertical rotation angle φ of the target drone, and the direction information of the suspected target drone in the actual space is obtained;

[0140] S440, the detection radar 21 detects objects within the range of three 120° fan-shaped areas connected end to end and within a distance of 50 meters, forming a target detection around, and then the detection radar 21 connects the RS485 bus to communicate with the edge computing terminal 4, and obtains information on the target distance, horizontal offset angle and vertical offset angle at the same time;

[0141] S450, the edge computing terminal 4 uses the coordinate information obtained by the method in the above step S420 to match the collected radar sensor information and the image sent back by the camera sensor, and further uses the micro-Doppler spectrum after the short-time Fourier transform STFT of the millimeter-wave radar echo data to filter out invalid information such as flying bird targets, and finally relies on the capture net launch device 33 to control the target drone;

[0142] S500, using the two-dimensional pan-tilt platform 32 to control the direction of the net-launching device 33. Since the net-launching device 33 is installed on a two-dimensional pan-tilt platform 32, it can be rotated to any horizontal and any pitch angle within a controllable range, so that the net-launching device 33 can face the target directly, so as to accurately capture the target drone;

[0143] S600: When the detection radar 21 detects that the distance of the target UAV is within the controllable range, the UAV capture net is automatically launched to launch the capture net operation on the target UAV in the detection area to achieve the purpose of UAV control.

[0144] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0145] Specific implementation scheme three: Different from specific implementation scheme two, in step S440, before transmitting back to the target sensor, when detecting the target drone, if the detection radar 21 uses a fixed threshold value to judge the echo, it cannot meet the actual complex air clutter environment. Therefore, constant false alarm probability (CFAR) detection is used to present the real drone target and further control the operation.

[0146] Among them, the optimal detection threshold T is:

[0147] T=-μ0 ln P fa (7)

[0148] Among them, μ0 is the clutter interference power;

[0149] The improved OS-CFAR algorithm is used, including sorting the reference cells from small to large, and using the sorted kth sample cell as background clutter interference.

[0150] The value of k should be multiplied as follows:

[0151] k=r1·N (8)

[0152] At this time, the false alarm probability is:

[0153]

[0154] Among them, r1 is the custom ratio, Γ represents the factorial; k() is the number of combinations; N is the reference unit length; α is the threshold factor; n is the total number of sample units. Through this constant false alarm probability algorithm, the anti-interference ability against invalid targets can be improved in the multi-target scenario in the air.

[0155] Specific implementation scheme four: Different from specific implementation scheme two, in step S450, when matching the data of the detection radar 21 with the image target recognition data transmitted back by the wide-angle camera sensor 23, the video frame data in RGB format transmitted back by the wide-angle camera sensor 23 obtains the target category information of the target UAV and the interference target, the coordinate position information of the camera coordinate system where the target is located, the pixel area size information occupied by the target, and the confidence through the deep learning model deployed in the edge computing terminal 4. The above-mentioned deep learning model is an existing model, and the method for obtaining the above-mentioned information is also an existing method; this information forms a one-dimensional vector, and each target is represented by a vector. At the same time, in order to match the radar sensor data, its coordinate information is transformed through step S420, so that the target coordinates obtained by the wide-angle camera sensor 23 and the target coordinates obtained by the detection radar 21 are in the same coordinate system;

[0156] This method adopts a progressive approach of time matching, space matching, and feature fusion. The detection radar transmits target data once every 50ms, that is, 20 sets of radar data per second, and the wide-angle camera sensor 23 transmits 30 frames of image information per second. Therefore, firstly, the radar target information is matched with the video frame of the wide-angle camera sensor 23, and the radar data is proportionally interpolated to obtain 30 sets of radar sensor data, and the acquisition time of the two sensors is synchronously matched; then the two sensor data obtained at the same time are further spatially matched, and the targets whose horizontal and vertical coordinates in the same coordinate system obtained before are within 3 pixels are merged; then a vector is used to represent the merged target. At this time, the vector contains both radar and visual sensor data, and finally the purpose of radar and visual fusion is achieved.

[0157] Specific implementation scheme five: Different from specific implementation scheme four, the following model can be used for coordinate transformation in step S320 to calibrate the camera sensor image and remove lens distortion, in step S420 to make the two sensor data point clouds in the same coordinate system during radar vision fusion, and in steps S500 and S600 to convert the camera coordinates into real-world orientation information to guide the two-dimensional gimbal to turn and launch the capture net:

[0158]

[0159] It should be noted that although the units of the above coordinates are all pixels, they do not affect the subsequent calculation of direction angles. At this time, K is the intrinsic parameter matrix of the wide-angle camera sensor, and f u 、f ν , u0, v0 are internal parameter coefficients; the coordinates of the suspected target drone [u i ,v i ] T Calculate the coordinates [X i ,Y i ,Z i ] T ; If the former is defined as p i , the latter is defined as P i , formula (10) can be simplified as:

[0160] P i =Z i K -1 p i (11).

[0161] Specific implementation scheme six: Different from specific implementation scheme five, the antennas of the millimeter wave radar sensor are arranged as sub-arrays, with half wavelength λ / 2 as intervals, and arranged in a rectangular manner; combined Figure 6 As shown, assuming that the sub-arrays are A, B, C, and D from left to right and from top to bottom, the detection radar 21 realizes the DOA estimation of the target through multiple groups of receiving antennas. In actual use, the target angle information is obtained by calculating the phase difference, which follows the following formula:

[0162]

[0163] Where θ is the target arrival angle, λ is the signal wavelength, is the phase difference between the two antennas, and d is the antenna spacing;

[0164] Assume the distance difference of the radar signal is Δ, θ represents the horizontal direction and φ represents the vertical direction, Σ represents the distance sum of the radar signal, and S represents the received signal function, then we have the following relationship:

[0165]

[0166] The angular error can be calculated by the following formula:

[0167]

[0168] Among them, Δβ is the horizontal azimuth error; Δε is the vertical pitch angle error;

[0169] The error is subtracted to improve the accuracy of the detection radar 21 in measuring the target in two horizontal and vertical directions, and accurate target azimuth information is obtained through radar return data.

[0170] Specific implementation scheme seven: Different from specific implementation scheme six, the working process of the detection radar 21 is that the signal synthesizer transmits a high-frequency modulated signal through the radio frequency antenna, the signal contacts the target drone fuselage, a part of the signal is reflected, and at the same time, the millimeter wave radar receiving antenna continuously receives the echo with a time delay reflected by the target drone, and the echo is mixed with the original transmission signal through the mixer to obtain an intermediate frequency signal, which is then used for digital signal processing;

[0171] The signal output by the signal synthesizer can be expressed as:

[0172]

[0173] The echo signal received by the receiver can be expressed as:

[0174]

[0175] In the formula,

[0176]

[0177] Among them, τ is the echo delay; R0 is the distance between the millimeter wave radar and the target UAV; ν is the Doppler velocity; is the distance-related time; t m is the time associated with Doppler velocity; μ is the slope of the Chirp signal; f c is the Chirp signal frequency; c represents the speed of light;

[0178] The intermediate frequency signal obtained by mixing the mixer is:

[0179]

[0180] Taking the derivative of the intermediate frequency signal, we get:

[0181]

[0182] where f i is the distance-related frequency; f d is the frequency associated with the Doppler velocity.

[0183] Specific implementation scheme eight: Different from specific implementation scheme seven, in the process of controlling the two-dimensional pan-tilt platform 32 in step S500, the edge computing terminal 4 outputs PWM signals with different duty cycles to the motor drive module with specific commutation signals, so that the brushless motor maintains a constant torque, and finally drives the motor of the FPV to maintain a specific angle;

[0184] In order to reduce the noise when the brushless motor rotates, a sinusoidal SPWM signal is used in this process; at the same time, a six-axis electronic gyroscope 34 is used to output real-time acceleration and angular velocity information, which is connected to the controller through an IIC communication interface; in addition, a PID algorithm is used to make the motor movement smoother, and the algorithm can be expressed as:

[0185] e(t)=y(t)-r(t) (20)

[0186]

[0187] Wherein, e(t) is the difference between the actual angle position r(t) calculated based on the data sent back by the electronic gyroscope 34 and the target angle position y(t); u(t) is the voltage output to the motor; u0 is the static error; K p , T1, T D It is a parameter adjusted according to the actual required response speed;

[0188] On this basis, sampling is performed with T as the sampling period, and the following is obtained:

[0189]

[0190] where e k is the deviation value input at the kth sampling time, k is the sampling sequence number, (k=0,1,2,…);

[0191] The evolved digital algorithm is more convenient for the controller to process, and the difference between the target angle and the actual angle is used as a factor for real-time feedback, thereby realizing the control of the brushless motor of the two-dimensional gimbal 32, so that the two-dimensional gimbal 32 can quickly respond to the target angle direction of the controller, and its facing direction points to the horizontal azimuth and pitch azimuth of the target drone, thereby improving the overall real-time performance;

[0192] In this process, the coordinates [X i ,Y i ,Z i ] T , and the relationship between the horizontal azimuth and pitch azimuth of the target UAV is:

[0193]

[0194] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A radar-vision fusion device for automatic capture of rotary-wing UAVs, characterized by: The invention comprises a top bracket (12) of a cross-country drone arranged above the cross-country drone, the top bracket (12) of the cross-country drone being provided with an edge computing terminal (4) and a drone controller (14), the top bracket (12) of the cross-country drone being provided with a front bracket (11) of the cross-country drone, the front bracket (11) of the cross-country drone being provided with a multi-sensor sensing component, the multi-sensor sensing component comprising a detection radar (21), a wide-angle camera sensor (23) and a radio frequency detection sensor (25), three detection radars (21) with the same parameters being distributed in a triangular shape and fixed on the front bracket (11) of the cross-country drone through a radar fixing seat (22), the radio frequency detection sensor (25) being arranged at the center of the detection radars (21) distributed in a triangular shape, and the side end of the front bracket (11) of the cross-country drone being connected with the wide-angle camera sensor (23) through a camera sensor connecting frame (24); A two-dimensional pan-tilt platform (32) is connected to the bottom of the drone via a bottom bracket (13) of the drone, a net-catching launch device (33) is provided at the moving end of the two-dimensional pan-tilt platform (32), and an electronic gyroscope (34) is provided on the net-catching launch device (33).

2. A method for the radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 1, characterized in that: The following steps are involved: S100, after an unauthorized target drone appears, the radio frequency detection sensor (25) detects the 2.4 GHz radio frequency signal communicated between the target drone and the remote controller controlling its flight, and obtains the airspace azimuth angle where the target drone is located; S200, the drone flies in the airspace direction of the target drone obtained in step S100, hovers near the target drone, and uses the detection radar (21) and the wide-angle camera sensor (23) to identify the target drone; S300, combining the image obtained by the wide-angle camera sensor (23) after dedistortion, and identifying the target drone through a pre-trained neural network model deployed on the edge computing terminal (4); S400, obtaining direction information of the target UAV through three-dimensional positioning, fusing the obtained direction information with the direction information obtained by the detection radar (21) through a series of coordinate transformations, and obtaining position information of the target UAV; S500, using the two-dimensional gimbal (32) to control the orientation of the net-capturing launching device (33), so that the net-capturing launching device (33) faces the target during the flight of the drone, so as to accurately capture the target drone; S600: When the detection radar (21) detects that the distance between the target UAV and the crossing aircraft enters a controllable range, the capture net is automatically launched to achieve the purpose of UAV control.

3. The method for the radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 2, characterized in that: In step S300, it includes: S310, calibrating the wide-angle camera sensor (23) to obtain the internal and external parameters and the distortion matrix of the wide-angle camera sensor (23); using a black and white chessboard calibration method and a Harris algorithm to detect corner points of black and white changes, wherein the detection function of the corner points is simplified to: Among them, w(x,y) is the window function; E(u,v) is the energy function; u,v represent the movement of the window; x,y are the pixel coordinates; I is the gray value function; M is a real symmetric matrix; S320, according to the internal and external parameters of the wide-angle camera sensor (23) obtained in step S310, coordinate transformation operation is performed on the panoramic image captured by the wide-angle camera sensor (23), which is decomposed into two steps: rotation transformation and translation transformation. The center of the wide-angle camera sensor (23) is set as the origin, and the vertical wide-angle camera sensor (23) is used to point outward as the Z axis to establish a left-handed system, which is defined as the camera coordinate system. The relationship between the camera coordinate system and the coordinate system of the actual space is as follows: in, is the coordinate in the camera coordinate system, is the coordinate in the actual space, R matrix is ​​the rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values ​​of the rotation angles in each direction; T is the translation transformation, i.e., the external parameter in step S310; The linear coordinates are expressed as: This forms the required camera coordinate system; S330, performing a dedistortion operation and establishing a nonlinear distortion model: Among them, a x ,a y ,u0,v0 are the internal parameters obtained in step S310; k1,k2,p1,p2 are distortion parameters; (u1,v1) are the coordinates after dedistortion; r is the Euclidean distance between two points; The eccentric distortion is: The radial distortion is: According to the actual de-distortion effect, the distortion model parameters are continuously optimized.

4. The method for the radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 3 is characterized in that: If the visibility is low, the echo data of the detection radar (21) is used to perform short-time Fourier transform (STFT) to obtain micro-Doppler information, and compared with the stored micro-Doppler data of the UAV and the interference target, so as to accurately identify the target as a UAV.

5. The method for the radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 2, characterized in that: In step S400, it includes: S410, the wide-angle camera sensor (23) transmits the RGB color information of the video screen in units of video frames, and transmits the information collected by the camera to the edge computing terminal (4), and the edge computing terminal (4) identifies a series of related frame signals and line signals of SoF and EoL therein; S420, set the i-th frame of the video image obtained in step S410 to I i , then the coordinates of the target drone in the camera coordinate system are defined as [X i ,Y i ,Z i ] T , the target detection model deployed on the edge computing terminal (4) obtains the pixel coordinates of the target drone as [u i ,v i ] T , coordinate transformation between two coordinate systems; S430: In step S420, the coordinates of the target UAV in the camera coordinate system are obtained [X i ,Y i ,Z i ] T Then, the coordinates are used to calculate the horizontal rotation angle θ and the vertical rotation angle φ of the target UAV, and the direction information of the target UAV in the actual space is obtained; S440, the detection radar (21) communicates with the edge computing terminal (4), and simultaneously obtains information on the target distance, horizontal offset angle, and vertical offset angle; S450, the edge computing terminal (4) matches the collected radar sensor information with the coordinate information obtained by the method in step S420 using the image sent back by the camera sensor, and uses the micro-Doppler spectrum after short-time Fourier transform (STFT) of the echo data of the detection radar (21) to filter out invalid information.

6. The method for the radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 5, characterized in that: In step S440, before the signal is transmitted back to the detection radar (21), a constant false alarm probability detection is adopted, and the optimal detection threshold T is: T=-μ0ln P fa (7) Among them, μ0 is the clutter interference power; The improved OS-CFAR algorithm is used, including sorting the reference cells from small to large, and using the sorted kth sample cell as background clutter interference. The value of k is determined by multiplying as follows: k=r1·N (8) At this time, the false alarm probability is: Among them, r1 is the custom ratio, Γ is the factorial, k() is the number of combinations, N is the reference unit length, α is the threshold factor, and n is the total number of sample units.

7. The method for radar-vision fusion of a radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 6, characterized in that: In step S450, when matching the data of the detection radar (21) with the image target recognition data transmitted back by the wide-angle camera sensor (23), the video frame data in RGB format transmitted back by the wide-angle camera sensor (23) obtains target category information of the target UAV and the interference target, coordinate position information of the target in the camera coordinate system, pixel area size information occupied by the target, and confidence through the deep learning model deployed in the edge computing terminal (4); the coordinate information is transformed through step S420 so that the target coordinates obtained by the wide-angle camera sensor (23) and the target coordinates obtained by the detection radar (21) are in the same coordinate system; For radar information acquired by the detection radar (21), the radar target information is matched with the video frame of the wide-angle camera sensor (23), and the radar data is proportionally interpolated to synchronize the acquisition times of the two sensors; then, the two sensor data at the same time are spatially matched, and the targets whose horizontal and vertical coordinates in the same coordinate system differ by less than 3 pixels are merged; then, a vector is used to represent the merged target, and at this time, the vector contains both radar and visual sensor data, thereby finally achieving the purpose of radar and visual fusion.

8. The method for radar-vision fusion of a radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 7, characterized in that: When performing the coordinate transformation of step S320 to calibrate the camera sensor image, making the two sensor data point clouds in the same coordinate system during the radar vision fusion process of step S420, and converting the camera coordinates into real-world orientation information to guide the two-dimensional gimbal (32) to turn and launch the net in steps S500 and S600, the following model is used for coordinate transformation: Among them, K is the intrinsic parameter matrix of the wide-angle camera sensor, f u 、f ν , u0, v0 are internal parameter coefficients; Will [u i ,v i ] T Defined as p i , and [X i ,Y i ,Z i ] T Defined as P i , simplifying formula (10) to: P i =Z i K -1 p i (11)。 9. The method for radar-vision fusion of a radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 2, characterized in that: The antennas of the detection radar (21) are arranged as sub-arrays at intervals of half a wavelength λ / 2 in a rectangular arrangement; assuming that the sub-arrays are A, B, C, and D from left to right and from top to bottom, the detection radar (21) estimates the DOA of the target through multiple groups of receiving antennas, and calculates the phase difference to obtain the target angle information: Where θ is the target arrival angle, λ is the signal wavelength, is the phase difference between the two antennas, and d is the antenna spacing; Assume that the distance difference of the radar signal is Δ, θ is the horizontal direction, φ is the vertical direction, Σ is the sum of the distances of the radar signal, and S is the received signal function, then we have the following relationship: Among them, Δ θ is the radar signal path difference in the horizontal direction; Δ φ is the radar signal path difference in the vertical direction; The angular error is calculated by the following formula: Among them, Δβ is the horizontal azimuth error; Δε is the vertical pitch angle error; The error is subtracted to improve the accuracy of the detection radar (21) in measuring the target in two horizontal directions and vertical directions, and accurate target position information is obtained through radar return data; When the detection radar (21) is working, the signal contacts the fuselage of the target UAV, and a part of it is reflected. At the same time, the millimeter wave radar receiving antenna continuously receives the echo with a time delay reflected by the target UAV. The echo is mixed with the original transmission signal through a mixer to obtain an intermediate frequency signal, which is then used for digital signal processing; The signal output by the signal synthesizer is: The echo signal received by the receiver is: In the formula, Among them, τ is the echo delay; R0 is the distance between the millimeter wave radar and the target UAV; ν is the Doppler velocity; is the distance-related time; t m is the time associated with Doppler velocity; μ is the slope of the Chirp signal; f c is the frequency of Chirp signal; c is the speed of light; The intermediate frequency signal obtained by mixing the mixer is: Taking the derivative of the intermediate frequency signal, we get: where f i is the distance-related frequency; f d is the frequency associated with the Doppler velocity.

10. The method for radar-vision fusion of a radar-vision fusion device for automatic capture of a rotary-wing UAV according to claim 2, characterized in that: In the process of controlling the two-dimensional pan-tilt platform (32) in step S500, the edge computing terminal (4) outputs PWM signals with different duty cycles to the motor drive module to provide a specific commutation signal, so that the brushless motor maintains a constant torque, and finally drives the motor of the FPV to maintain a specific angle; A sine wave SPWM signal and a six-axis electronic gyroscope (34) are used to output real-time acceleration and angular velocity information through the electronic gyroscope (34); a PID algorithm is used to make the motor movement smoother: e(t)=y(t)-r(t) (20) Wherein, e(t) is the difference between the actual angle position r(t) calculated based on the data sent back by the electronic gyroscope (34) and the target angle position y(t); u(t) is the voltage output to the motor; u0 is the static error; K p , T1, T D It is a parameter adjusted according to the actual required response speed; Taking T as the sampling period for sampling, we get: Among them, e k is the deviation value input at the kth sampling time, k is the sampling sequence number, (k=0,1,2,…); The coordinates of the detected target drone camera coordinate system [X i ,Y i ,Z i ] T , and the relationship between the horizontal azimuth and pitch azimuth of the target UAV is:

Images