Lightning-vision fusion device and method for intelligent capture system of unmanned aerial vehicle

Abstract

The invention discloses a thunder-vision fusion device and method for an unmanned aerial vehicle intelligent capture system, belongs to the technical field of unmanned aerial vehicle high-altitude control, and solves the problems that an existing unmanned aerial vehicle control mode is high in recognition difficulty and not comprehensive in recognized orientation. The vehicle-mounted unmanned aerial vehicle carries a sound wave generating device, target identification is performed on a target unmanned aerial vehicle through a model deployed in a calculation unit on an unmanned aerial vehicle carrier, an interference signal transmitting device is turned to the direction of the target unmanned aerial vehicle through a rotating platform, a sound wave signal is transmitted, and management and control of the target unmanned aerial vehicle are performed; the communication vehicle capable of performing long-distance movement is used as an unmanned aerial vehicle carrier, so that the unmanned aerial vehicle can follow the communication vehicle to expand the movement range without being limited by regions, the operation range of the unmanned aerial vehicle is expanded, remote control and monitoring of the unmanned aerial vehicle can be realized without getting off the vehicle in the communication vehicle, and information returned by the unmanned aerial vehicle can be received in real time; the flexibility is high, and the endurance time is long.

Description

Technical Field

[0001] The present invention belongs to the technical field of high-altitude control of unmanned aerial vehicles, and more specifically, relates to a radar-vision fusion device and method for an intelligent capture system of unmanned aerial vehicles. Background Art

[0002] In recent years, drone technology has made significant progress, its performance has been continuously improved, and its application areas have become increasingly extensive. Drones have been widely used in many fields such as agricultural spraying, aerial photography, urban monitoring, emergency rescue, and logistics distribution. However, there are many difficulties in achieving precise control of drones. Drones fly in the air with high maneuverability and flexibility, which makes tracking and positioning particularly difficult. In addition, occlusion may occur, such as when drones are blocked by buildings, trees, etc., which will cause the tracking algorithm to lose complete information about the target and make it difficult to maintain the continuity and accuracy of tracking.

[0003] There are still some problems with the existing UAV synchronous positioning, tracking and control systems. For example, patent application number: CN202311338360.8 discloses an omnidirectional UAV autonomous identification and tracking system and method. This patent only tracks the target by controlling the rotation of the gimbal. The tracking range is limited and lacks flexibility. It uses a binocular camera for ranging. The binocular camera has higher requirements for the accuracy of calibration parameters, so the maintenance cost is very high in actual operation. For another example, patent application number 202420392718.9 discloses a UAV capture net launch device. In this technical solution, the capture net is thrown to the UAV through a launch device. The cost is high, which increases the launch cost, and the capture net needs to be removed later, which increases the workload. Summary of the invention

[0004] The present invention aims at solving the technical problems existing in the prior art and provides a radar-vision fusion device and method for an intelligent capture system of an unmanned aerial vehicle.

[0005] In order to solve the above technical problems, the technical solutions provided by the present invention are as follows: A radar-vision fusion device for an intelligent capture system of an unmanned aerial vehicle comprises a mounting bracket arranged on the unmanned aerial vehicle, a detection radar is arranged on the mounting bracket, an unmanned aerial vehicle carrier is arranged below the detection radar, an acoustic detection array is arranged on the unmanned aerial vehicle carrier, the unmanned aerial vehicle carrier transmits information with the unmanned aerial vehicle through an information transmission mechanism, a rotating platform is installed on the mounting bracket, and a sensor perception component and an interference signal transmitting device are connected to the rotating platform.

[0006] Preferably, the UAV carrier is a vehicle carrier, and the sound measurement array adopts a spherical microphone matrix; a computing unit is provided in the vehicle carrier, and the information transmission mechanism is connected to the computing unit; The mounting bracket is provided with three detection radars distributed in a triangle. The sensor sensing component is provided with an omnidirectional camera sensor. The omnidirectional camera sensor includes a plurality of wide-angle cameras arranged along the circumferential direction. The interference signal transmitting device is arranged at the upper end of the omnidirectional camera sensor.

[0007] The present invention also provides a radar-visual fusion method for an unmanned aerial vehicle intelligent capture system, comprising the following steps: S100, when an unauthorized target drone appears in a no-fly airspace, the acoustic array monitors the unique noise generated by the target drone during flight and transmits it back to the computing unit in the communication vehicle. After processing by the computing unit, the airspace azimuth angle where the target drone is located is obtained; S200, the vehicle-mounted UAV flies toward the airspace direction of the target UAV obtained in step S100, hovers near the target UAV, and uses detection radar and omnidirectional camera sensors to identify the target UAV; S300, combining the image obtained by the omnidirectional camera sensor after dedistortion, and identifying the target drone through the pre-trained neural network model deployed in the computing unit; S400, obtaining direction information of the target UAV through three-dimensional positioning, fusing the obtained direction information with the direction information obtained by the detection radar through a series of coordinate transformations, and obtaining position information of the target UAV; S500, causing the vehicle-mounted UAV to fly toward the target UAV, and using the rotating platform to control the jamming signal transmitter to adjust the direction toward the target UAV, so as to accurately capture the target UAV; S600: When the detection radar detects that the target UAV and the vehicle-mounted UAV are within the controllable range, the vehicle-mounted UAV uses the jamming signal transmitting device to transmit a sonic signal toward the target UAV to achieve the goal of shooting down the UAV.

[0008] Preferably, in step S300, it includes: S310, calibrate the omnidirectional camera sensor to obtain the internal and external parameters and distortion matrix of the omnidirectional camera sensor; use the Shi-Tomasi algorithm to detect the corner points of black and white changes, where the detection function of the corner points is simplified to:

[0009] In the formula, is the window function; is the energy function; To reflect the movement of the window; is the pixel coordinate; is the gray value function; is a real symmetric matrix; S320, according to the internal and external parameters of the omnidirectional camera sensor obtained in step S310, perform coordinate transformation operation on the panoramic image captured by the omnidirectional camera sensor, and use a rotation matrix and a translation vector to represent it. The rotation matrix is ​​responsible for the alignment of the coordinate axis direction, and the translation vector is responsible for the alignment of the origin. The center of the omnidirectional camera sensor is set as the origin, and the vertical omnidirectional camera sensor is pointed outward. Z Axis, establish a coordinate system, define it as the camera coordinate system, then the relationship between the camera coordinate system and the coordinate system of the actual space is as follows:

[0010] In the formula, is the coordinate in the camera coordinate system; is the coordinate in the real space; R The matrix is ​​a rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values ​​of the rotation angles in each direction; T is the translation transformation amount, i.e., the external parameter in step S310; The linear coordinates are expressed as:

[0011] This forms the required camera coordinate system; S330, performing a dedistortion operation and establishing a nonlinear distortion model:

[0012] In the formula, is the internal reference obtained in step S310; is the distortion parameter; is the coordinate after dedistortion; r is the Euclidean distance between two points; The radial distortion is: ; S340, perform optical axis correction to make the optical axes of the camera array parallel after correction. According to the imaging principle of a single camera, correct the world coordinates of any point in the front and back space. I The corresponding image coordinates i The mapping relationships are:

[0013] In the formula, c is the coordinate of the optical center in the world coordinate system; s is the projection of the distance from the object point to the optical center on the optical axis; A is the internal parameter matrix; R is the rotation matrix; the optical center remains unchanged before and after correction, that is c 1 and c 2 equal; After correction, the optical axis is parallel, which can be based on:

[0014] Get the relationship between the optical center position before and after correction: ; S350, after the image array is calibrated, the size of the overlapping area is calculated according to the same edge corner point coordinates; the length of the overlapping area of ​​the images captured by camera 1 and camera 2 L h for:

[0015] In the formula, is the length of the image captured by camera 1; is the edge corner point in the image captured by camera 1 The horizontal axis of Laser spot in the image captured by camera 2 The horizontal axis of The image stitching between camera 2 and camera 3, camera 3 and camera 4, and camera 4 and camera 5 is calculated using the same method.

[0016] Preferably, in step S400, it includes: S410, transmitting the image information collected by the omnidirectional camera sensor to the computing unit through the optical fiber, and the computing unit identifies a series of related frame signals and line signals of SoF and EoL therein; S420, the video image obtained in step S410 is i Frame Set to I i , then the coordinates of the target UAV in the camera coordinate system are defined as [ X i , Y i , Z i ] T , the target detection model deployed on the computing unit obtains the pixel coordinates of the target drone as [ u i , v i ] T , coordinate transformation between two coordinate systems; S430, in step S420, the coordinates of the target UAV in the camera coordinate system are obtained [ X i , Y i , Z i ] TThen, use this coordinate to calculate the azimuth of the target drone θ and pitch angle 𝜙, to obtain the direction information of the target UAV in the actual space; S440, the detection radar communicates with the computing unit, obtains information on the target distance, azimuth, and pitch angle, and performs coordinate conversion; S450, the computing unit matches the collected radar sensor information with the coordinate information obtained by the method in step S420 of the image returned by the camera sensor, and obtains the difference frequency signal for detection using the information returned by the echo data of the detection radar.

[0017] Preferably, in step S440, if the visibility is low, before the signal is transmitted back to the detection radar, the background clutter power estimate is calculated by averaging the reference cells on both sides using the improved CA-CFAR algorithm, assuming that the target exists:

[0018] In the formula, X i It is a protection unit. Y i is the reference unit, n is the total number of unilateral reference units; The detection threshold after obtaining the adjusted background power estimate is:

[0019] in, α is the threshold factor, which determines the level of the detection threshold relative to the background power.

[0020] Preferably, in step S440, the detection radar acquires the data set to perform coordinate conversion for data comparison, converts the millimeter wave radar coordinates into world coordinates, and P ( β , Q ) is in world coordinates:

[0021] In the formula, the distance between the XOY planes of the two coordinate systems is Z 0 , the XOZ plane distance is Y w .

[0022] Preferably, the detection radar sends a detection signal to the target UAV, and the target receives the signal and returns an echo signal to the millimeter wave radar. The transmission signal of the sawtooth frequency modulation continuous wave radar can be expressed as:

[0023] In the formula, represents the initial phase, represents the radar signal amplitude, Indicates the starting frequency, It represents the frequency modulation slope, and its formula is as follows:

[0024] In the formula, B Indicates the frequency bandwidth, T Indicates the frequency modulation repetition period; For a strong scattering point target with a distance of R and a velocity of v, its echo signal is expressed as follows:

[0025] In the formula, C Indicates the amplitude attenuation of the signal; The sawtooth radar transmit signal is used as the local oscillator signal, and the difference frequency signal is formed by mixing the mixer and the echo. S B : ; From this, the frequency expression of the difference frequency signal is deduced: ; It can be concluded that: Center frequency ; FM slope ; FM Phase .

[0026] Preferably, the detection radar uses the DBSCAN-CFSFDP clustering algorithm when processing data, and the local density information entropy formula is:

[0027] In the formula, i Indicates the data set i data points, 1 b represents the logarithm with base 2, K Indicates that under a given neighborhood radius, i The ratio of the number of data points within the radius to the total number of points, that is i Probability distribution of local density of points; Then consider the distribution of the entire data set and average the local density information entropy of all data points, which is expressed as:

[0028] In the formula, n is the number of data points in the dataset; Epsis the neighborhood radius, using the determined Eps Parameters, determine the noise points based on the number of data points in the neighborhood, remove the noise points, and obtain clean point cloud data; Select the starting point corresponding to the most stable information entropy change Eps , the density distribution between data points in the stable segment is relatively consistent; the Gini coefficient is used to evaluate the uneven distribution of data under different truncation distances and determine the optimal truncation distance best _ d c , and using the local density ρ i and the relative distance δ i The product of is used to make a descending graph of cluster center weights, and an adaptive exponential decay function curve is introduced to automatically determine the cluster center point; Local density: ; Clarity of CFSFDP algorithm clustering:

[0029] In the formula, P i For the i The ratio of the cluster center weight of a point to the total cluster center weight is expressed as follows:

[0030] In the formula, y i Indicates i The cluster center weight of each point is expressed as: .

[0031] Preferably, the decision-level PDA is used to fuse the point cloud and image detection results when fusing radar and camera data:

[0032] In the formula, S is the area of ​​the target detection box, i The number of pixels that the target detection box occupies in the image. The pixel value of each pixel will affect the size of the detection box. Project the 3D detection box generated by the radar onto the plane and treat it as a simplified rectangular box:

[0033] In the formula, Y , Z Respectively represent the y-axis and z-axis coordinates of the detection box vertex on the two-dimensional plane; The basis for determining the overlap is set as the comparison value of the overlap area with the lidar detection frame and the camera detection frame area. The overlap can be used to accurately determine the degree of match between the lidar detection results and the camera detection results. The overlap formula is:

[0034] In the formula, μ is the overlap of two detection frames, and s μ represents their overlapping area.

[0035] Compared with the prior art, the present invention has the following beneficial effects: The present invention adopts a method of carrying a sound wave generating device on a vehicle-mounted drone, and performs target identification on the target drone through a control device on the drone carrier, and turns the interference signal transmitting device to the direction of the target drone through a rotating platform, and transmits a sound wave signal to control the target drone; with the help of a drone carrier that can move over long distances, the drone can follow the drone carrier to expand its moving range and is no longer restricted by the region, thereby expanding the operating range of the drone, and remote control and monitoring of the drone can be achieved in the drone carrier without getting off the vehicle, and information sent back by the drone can be received in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 It is a schematic diagram of the three-dimensional structure of an embodiment of a radar-visual fusion device for an intelligent capture system of a UAV according to the present invention; Figure 2 It is a flow chart of an embodiment of a radar-visual fusion method for an intelligent capture system of a UAV according to the present invention; Figure 3 A schematic diagram of the coordinate system transformation relationship of an embodiment of the present invention; Figure 4 This is a picture collected by an omnidirectional camera sensor in an embodiment of the present invention; Figure 5 It is a schematic diagram of the decision-level fusion principle of the millimeter-wave radar and the omnidirectional camera sensor in an embodiment of the present invention; Figure 6 Schematic diagram of sensor fusion in an embodiment of the present invention.

[0038] Explanation of symbols in the figure: 1. Detection radar; 2. UAV carrier; 21. Communication vehicle; 3. Acoustic array; 4. Information transmission mechanism; 41. Transmission optical fiber; 5. UAV; 6. Rotating platform; 7. Interference signal transmitter; 8. Computing unit; 9. Top mounting bracket; 10. Bottom mounting bracket; 11. Radar fixing seat; 12. Camera mounting bracket; 13. All-round camera sensor; 14. Mounting column; 15. Interference signal transmitter chassis. DETAILED DESCRIPTION

[0039] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the following is a further detailed description of a radar-visual fusion device and method for a drone intelligent capture system provided by the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] Example 1 See also Figure 1 The present embodiment provides a radar-vision fusion device for an intelligent capture system of a UAV, including a mounting bracket arranged on a UAV 5, a detection radar 1 being arranged on the mounting bracket, a UAV carrier 2 being arranged below the detection radar 1, a sound detection array 3 being arranged on the UAV carrier 2, the UAV carrier 2 transmitting information with the UAV 5 via an information transmission mechanism 4, a rotating platform 6 being installed on the mounting bracket, and a sensor sensing component and an interference signal transmitting device 7 being connected to the rotating platform 6.

[0041] The embodiment of the present invention adopts the method of carrying a sound wave generating device on a vehicle-mounted drone, and identifies the target drone through the control device on the drone carrier 2, and turns the interference signal transmitting device to the direction of the target drone through the rotating platform 6, and transmits the sound wave signal to control the target drone; with the help of the drone carrier 2 that can move over long distances, the drone 5 can follow the drone carrier 2 to expand the moving range, no longer be restricted by the area, and expand the operating range of the drone. It is possible to remotely control and monitor the drone 5 in the drone carrier 2 without getting off the vehicle, and can receive information sent back by the drone in real time.

[0042] In this embodiment, the drone carrier 2 is a vehicle carrier, and the vehicle carrier adopts a communication vehicle 21. A computing unit 8 is provided in the vehicle carrier, and the information transmission mechanism 4 is connected to the computing unit 8. The drone 5 is arranged directly above the communication vehicle 21.

[0043] Specifically, a trained neural network model is deployed in the computing unit 8, which can identify the target drone. The information transmission mechanism 4 is provided with a transmission optical fiber 41. Information is transmitted to the computing unit 8 in the communication vehicle 21 through the transmission optical fiber 41, and the information transmission is stable.

[0044] like Figure 1 As shown, the mounting bracket includes a top mounting bracket 9 and a bottom mounting bracket 10. The top mounting bracket 9 is arranged on the upper part of the UAV 5, and the bottom mounting bracket 10 is arranged on the bottom of the UAV 5. The detection radar 1 is installed on the bottom mounting bracket 10, and three detection radars 1 with the same parameters are arranged on the bottom mounting bracket 10. The three detection radars 1 are distributed in a triangle, and the detection radar 1 is fixedly mounted on the bottom mounting bracket 10 through a radar fixing seat 11.

[0045] In this embodiment, the rotating platform 6 is fixedly mounted on the top mounting bracket 9, the sensor sensing component is mounted on the top mounting bracket 9 at the upper end of the rotating platform 6 through a camera mounting bracket 12, and the sensor sensing component is provided with an omnidirectional camera sensor 13; the interference signal transmitting device 7 is mounted on the upper end of the omnidirectional camera sensor 13 through four mounting columns 14 and the interference signal transmitting device chassis 15.

[0046] Furthermore, the omnidirectional camera sensor 13 is composed of four wide-angle cameras evenly arranged along the circumferential direction, each camera has a detection range of 120°, and the four cameras are camera 1, camera 2, camera 3 and camera 4; the omnidirectional camera sensor 13 cooperates with the detection radar 1 to identify the target drone in the target airspace and can collect images. The rotating platform 6 can rotate up and down in a range of 0-60°. The interference signal transmitter 7 located on it can be turned toward the direction of the target drone through the rotating platform 6; the interference signal transmitter 7 can emit a 2.4Ghz clutter interference signal, which can emit interference sound wave signals to the identified target drone to control the target drone.

[0047] Furthermore, as a preferred embodiment of the present invention, the sensor perception component is also provided with an infrared detection sensor, which obtains preliminary position information of the target UAV by detecting infrared radiation released by the target UAV, so that the vehicle-mounted UAV approaches the target UAV; then the images taken around are transmitted to the computing unit 8 through the omnidirectional camera sensor 13, and the detection radar 1 cooperates to detect objects within the effective distance around and sends the position information to the computing unit 8. The computing unit 8 performs coordinate transformation and time synchronization to realize radar-vision fusion, and obtains accurate position information of the target UAV after eliminating interference objects, so as to control it.

[0048] In this embodiment, the sound measuring array 3 is arranged at the upper front end of the communication vehicle 21. The sound measuring array 3 adopts a spherical microphone matrix (SMA). The sound measuring array 3 is used to locate the sound direction of the target area and identify the noise source. By analyzing the received sound signal, the source of the noise can be quickly identified and preliminarily located.

[0049] The detection radar 1 adopts a frequency modulated continuous wave (FMCW) radar. Three detection radars 1 distributed in a triangle form a radar detection system. Each detection radar can detect a range of 120°.

[0050] Furthermore, the communication vehicle 21 is equipped with an onboard high-power generator and an uninterruptible power supply (UPS), which can extend the flight time of the UAV.

[0051] Furthermore, as a preferred embodiment of the present invention, the number of detection radars 1 and the number of cameras specifically arranged on the mounting bracket can be adaptively arranged according to the actual use environment.

[0052] Furthermore, as a preferred embodiment of the present invention, the vehicle carrier is used as a mobile device for capturing the drone. The specific vehicle form can be adapted according to the actual use environment and the type of drone carried. At the same time, other movable vehicles can also be used as mobile carriers for the drone. The communication vehicle 21 has long-distance mobility, so that the drone can follow the communication vehicle to expand the range of movement, no longer restricted by the region, and expand the operation range of the drone. The drone can be remotely controlled and monitored in the communication vehicle without getting off the vehicle, and the information sent back by the drone can be received in real time.

[0053] The embodiment of the present invention first obtains the approximate direction information of the target drone through the acoustic array 3, so that the vehicle-mounted drone is launched. Then, the omnidirectional camera sensor 13 and the detection radar 1 start working, and the target and its position information are obtained through the detection radar 1. The omnidirectional camera sensor 13 recognizes the image and transmits the obtained image to the computing unit 8. The computing unit 8 identifies the real drone target, eliminates interference such as insects and birds, and cooperates with the interference signal transmitting device 7 to transmit high-intensity electromagnetic waves to block the link, thereby interfering with the target and finally achieving the goal of shooting down the drone.

[0054] Example 2 The embodiment of the present invention provides a radar-visual fusion method for an unmanned aerial vehicle intelligent capture system, comprising the following steps: S100. When an unauthorized target drone appears in a no-fly airspace, the acoustic array 3 monitors the unique noise generated by the target drone during flight and transmits it back to the computing unit 8 in the communication vehicle 21. After processing by the computing unit 8, the approximate azimuth angle of the airspace where the target drone is located is obtained.

[0055] S200, the vehicle-mounted UAV flies toward the airspace direction of the target UAV obtained in step S100, hovers near the target UAV, and uses the detection radar 1 and the omnidirectional camera sensor 13 to identify the target UAV.

[0056] S300 , combining the image obtained by the omnidirectional camera sensor 13 after dedistortion, and identifying the target drone through a pre-trained neural network model deployed in the computing unit 8 .

[0057] The specific steps include the following: S310, calibrate the omnidirectional camera sensor 13 to obtain the internal and external parameters and distortion matrix of the omnidirectional camera sensor 13; use the Shi-Tomasi algorithm to detect the corner points of black and white changes, where the detection function of the corner points is simplified to:

[0058] In the formula, is the window function; is the energy function; To reflect the movement of the window; is the pixel coordinate; is the gray value function; is a real symmetric matrix; S320, according to the internal and external parameters of the omnidirectional camera sensor 13 obtained in step S310, coordinate transformation operation is performed on the panoramic image captured by the omnidirectional camera sensor 13, which can be represented by a rotation matrix and a translation vector. The rotation matrix is ​​responsible for the alignment of the coordinate axis direction, and the translation vector is responsible for the alignment of the origin. The center of the omnidirectional camera sensor is set as the origin, and the vertical omnidirectional camera sensor is pointed outward. Z Axis, establish a coordinate system, define it as the camera coordinate system, then the relationship between the camera coordinate system and the coordinate system of the actual space is as follows:

[0059] In the formula, is the coordinate in the camera coordinate system; is the coordinate in the real space; R The matrix is ​​a rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values ​​of the rotation angles in each direction; T is the translation transformation amount, i.e., the external parameter in step S310; The linear coordinates are expressed as:

[0060] This forms the required camera coordinate system; S330, performing a dedistortion operation and establishing a nonlinear distortion model:

[0061] In the formula, is the internal reference obtained in step S310; is the distortion parameter; is the coordinate after dedistortion; ris the Euclidean distance between two points; The radial distortion is: ; S340, perform optical axis correction to make the optical axes of the camera array parallel after correction. According to the imaging principle of a single camera, correct the world coordinates of any point in the front and back space. I The corresponding image coordinates i The mapping relationships are:

[0062] In the formula, c is the coordinate of the optical center in the world coordinate system; s is the projection of the distance from the object point to the optical center on the optical axis; A is the internal parameter matrix; R is the rotation matrix; the optical center remains unchanged before and after correction, that is c 1 and c 2 equal; After correction, the optical axis is parallel, which can be based on:

[0063] Get the relationship between the optical center position before and after correction: ; S350, after the image array is calibrated, the size of the overlapping area is calculated according to the same edge corner point coordinates; the length of the overlapping area of ​​the images captured by camera 1 and camera 2 L h for:

[0064] In the formula, is the length of the image captured by camera 1; is the edge corner point in the image captured by camera 1 The horizontal axis of Laser spot in the image captured by camera 2 The horizontal axis of The image stitching between camera 2 and camera 3, camera 3 and camera 4, and camera 4 and camera 5 is calculated using the same method.

[0065] S400, obtaining direction information of the target UAV through three-dimensional positioning, and fusing the obtained direction information with the direction information obtained by the detection radar 1 through a series of coordinate transformations to obtain the position information of the target UAV.

[0066] The specific steps include the following: S410, transmitting the image information collected by the omnidirectional camera sensor 13 to the computing unit 8 via an optical fiber, and the computing unit 8 identifies a series of related frame signals and line signals of SoF and EoL therein; S420, the video image obtained in step S410 is i Frame Set to I i , then the coordinates of the target UAV in the camera coordinate system are defined as [ X i , Y i , Z i ] T , the target detection model deployed on the computing unit obtains the pixel coordinates of the target drone as [ u i , v i ] T , coordinate transformation between two coordinate systems; S430, in step S420, the coordinates of the target UAV in the camera coordinate system are obtained [ X i , Y i , Z i ] T Then, use this coordinate to calculate the azimuth of the target drone θ and pitch angle 𝜙, to obtain the direction information of the target UAV in the actual space; S440, the detection radar 1 communicates with the calculation unit 8, obtains information on the target distance, azimuth, and pitch angle, and performs coordinate conversion; S450, the calculation unit 8 matches the collected radar sensor information with the coordinate information obtained by the method in step S420 of the image returned by the camera sensor, and obtains the difference frequency signal for detection using the information returned by the echo data of the detection radar.

[0067] Furthermore, when performing step S440, if the visibility is low, before the signal is transmitted back to the detection radar, assuming that the target exists, the improved CA-CFAR algorithm is used to calculate the background clutter power estimate by averaging the reference cells on both sides:

[0068] In the formula, X i It is a protection unit. Y i is the reference unit, n is the total number of unilateral reference units; The detection threshold after obtaining the adjusted background power estimate is:

[0069] in, α is the threshold factor, which determines the level of the detection threshold relative to the background power. By reducing the background power estimate affected by multiple nearby targets, the detection threshold can be reduced accordingly, thereby improving the probability of target detection in these complex environments.

[0070] Furthermore, when performing step S440, the detection radar 1 obtains the data set to perform coordinate conversion for data comparison, converts the millimeter wave radar coordinates into world coordinates, and P ( β , Q ) is in world coordinates:

[0071] In the formula, the distance between the XOY planes of the two coordinate systems is Z 0 , the XOZ plane distance is Y w .

[0072] Furthermore, when matching the data of the detection radar 1 with the image target recognition data returned by the omnidirectional camera sensor 13, the video frame data in BRG format returned by the omnidirectional camera sensor 13 obtains the target category information of the target UAV and the interference target, the coordinate position information of the target in the camera coordinate system, the pixel area size information occupied by the target, and the confidence level through the deep learning model deployed in the computing unit 8; the coordinate information is transformed through step S420 so that the target coordinates obtained by the omnidirectional camera sensor 13 and the target coordinates obtained by the detection radar 1 are in the same coordinate system.

[0073] For the radar information obtained by the detection radar 1, the radar target information is matched with the video frame of the omnidirectional camera sensor 13, and the camera image and the point cloud data of the detection radar are synchronized through the timestamp in the ROS system to ensure that the data are at the same time. Then, the image and point cloud are input into the corresponding target detection algorithm respectively, and the two-dimensional image detection frame and the three-dimensional point cloud detection frame are obtained, and the detection frame is projected to the image coordinate system. Finally, the overlap of the projected detection frame is calculated, and the result is processed according to the fusion strategy.

[0074] Furthermore, in this embodiment, the detection radar 1 sends a detection signal to the target drone, and the target receives the signal and returns an echo signal to the millimeter wave radar. The transmission signal of the sawtooth frequency modulation continuous wave radar can be expressed as:

[0075] In the formula, represents the initial phase, represents the radar signal amplitude, Indicates the starting frequency, It represents the frequency modulation slope, and its formula is as follows:

[0076] In the formula, B Indicates the frequency bandwidth, T Indicates the frequency modulation repetition period; For a strong scattering point target with a distance of R and a velocity of v, its echo signal is expressed as follows:

[0077] In the formula, C Indicates the amplitude attenuation of the signal; The sawtooth radar transmit signal is used as the local oscillator signal, and the difference frequency signal is formed by mixing the mixer and the echo. S B : ; From this, the frequency expression of the difference frequency signal is deduced: ; It can be concluded that: Center frequency ; FM slope ; FM Phase .

[0078] Furthermore, in this embodiment, the detection radar 1 adopts the DBSCAN-CFSFDP clustering algorithm when processing data. The clustering algorithm first proposes local density information entropy to obtain the optimal neighborhood radius parameter for the DBSCAN algorithm.

[0079] The local density information entropy formula is:

[0080] In the formula, i Indicates the data set i data points, 1 b represents the logarithm with base 2, K Indicates that under a given neighborhood radius, i The ratio of the number of data points within the radius to the total number of points, that is i Probability distribution of local density of points; Then consider the distribution of the entire data set and average the local density information entropy of all data points, which is expressed as:

[0081] In the formula, n is the number of data points in the dataset; Eps is the neighborhood radius, using the determined Eps Parameters, determine the noise points based on the number of data points in the neighborhood, remove the noise points, and obtain clean point cloud data.

[0082] Select the starting point corresponding to the most stable information entropy change Eps , the density distribution between data points in the stable segment is relatively consistent, making it easier to classify scattered noise points separately; the Gini coefficient is used to evaluate the uneven distribution of data under different truncation distances and determine the optimal truncation distance best _ d c , and using the local density ρ i and the relative distance δ i The product of is used to make a descending graph of cluster center weights, and an adaptive exponential decay function curve is introduced to automatically determine the cluster center points.

[0083] Local density: ; Clarity of CFSFDP algorithm clustering:

[0084] In the formula, P i For the i The ratio of the cluster center weight of a point to the total cluster center weight is expressed as follows:

[0085] In the formula, y i Indicates i The cluster center weight of each point is expressed as: .

[0086] Furthermore, when fusing radar and camera data, a decision-level PDA is used to fuse point cloud and image detection results:

[0087] In the formula, S is the area of ​​the target detection box, i It is the number of pixels occupied by the target detection box in the image. The pixel value of each pixel will affect the size of the detection box.

[0088] In this embodiment, the detection radar 1 may detect irregular objects, which may cause the detection frame on the projected image to present an irregular image. In order to facilitate comparison and a series of calculations, the three-dimensional detection frame generated by the radar is projected onto a plane and treated as a simplified rectangular frame. This simplified processing method makes area calculation more convenient and facilitates the comparison between radar and camera detection results. In this way, the comparison of laser radar and camera detection results can be more intuitive.

[0089]

[0090] In the formula, Y , Z They represent the y-axis and z-axis coordinates of the detection box vertices on the two-dimensional plane respectively.

[0091] The basis for determining the overlap is set as the comparison value of the overlap area with the lidar detection frame and the camera detection frame area. The overlap can be used to accurately determine the degree of match between the lidar detection results and the camera detection results. The overlap formula is:

[0092] In the formula, μ is the overlap of two detection frames, and s μ represents their overlapping area.

[0093] S500, the vehicle-mounted UAV is made to fly toward the target UAV, and the interference signal transmitting device 7 is controlled by the rotating platform 6 to adjust the direction toward the target UAV, so as to accurately capture the target UAV.

[0094] S600: When the detection radar 1 detects that the target UAV and the vehicle-mounted UAV are within the controllable range, the vehicle-mounted UAV uses the interference signal transmitting device 7 to transmit a sound wave signal toward the target UAV to achieve the goal of shooting down the UAV.

[0095] The present invention provides a radar-visual fusion device and method for an intelligent capture system of a drone, which solves the problems of limited tracking range, lack of flexibility, difficulty in identifying features of drones, and short endurance of existing drones. The present invention first obtains the approximate direction information of the target drone through an acoustic array 3, so that the vehicle-mounted drone is lifted off and approaches the target drone. Subsequently, the omnidirectional camera sensor 13 and the detection radar 1 start to work, and the detection radar 1 begins to identify objects within the effective distance around and transmits them to the calculation unit 8. The target and its position information are obtained through the detection radar 1. The omnidirectional camera sensor 13 identifies the image and transmits the obtained image to the calculation unit 8. The calculation unit 8 identifies the real drone target, and the calculation unit 8 performs coordinate transformation and time synchronization to realize radar-visual fusion. After eliminating interference objects such as insects and birds, the accurate position information of the target drone is obtained, and the interference signal transmitting device 7 is used to transmit high-intensity electromagnetic waves to block the link, thereby interfering with the target, and finally achieving the goal of shooting down the drone and controlling the target airspace.

[0096] The invention has comprehensive identification direction, sensitive system and stable signal transmission. The invention adopts the method of carrying a sound wave generating device on a vehicle-mounted unmanned aerial vehicle, and performs target identification on the target unmanned aerial vehicle through a model deployed in a computing unit on the unmanned aerial vehicle carrier, and turns the interference signal transmitting device to the direction of the target unmanned aerial vehicle through a rotating platform, and transmits a sound wave signal to control the target unmanned aerial vehicle; with the help of a communication vehicle that can move over long distances as a unmanned aerial vehicle carrier, the unmanned aerial vehicle can follow the communication vehicle to expand its mobile range and is no longer restricted by the region, thereby expanding the operating range of the unmanned aerial vehicle, and remote control and monitoring of the unmanned aerial vehicle can be achieved in the communication vehicle without getting off the vehicle, and information sent back by the unmanned aerial vehicle can be received in real time, and the flexibility is high and the endurance time is long.

[0097] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0098] In addition, in the description of the present application, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0099] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A radar-visual fusion device for an intelligent capture system of a drone, comprising a mounting bracket arranged on the drone, the mounting bracket being provided with a detection radar, characterized in that: A drone carrier is provided below the detection radar, on which a sound detection array is provided. The drone carrier transmits information with the drone via an information transmission mechanism. A rotating platform is installed on the mounting bracket, on which a sensor sensing component and an interference signal transmitting device are connected.

2. According to claim 1, a radar-visual fusion device for an intelligent capture system of a UAV is characterized by: The UAV carrier is a vehicle carrier, and the sound detection array adopts a spherical microphone matrix; a computing unit is provided in the vehicle carrier, and the information transmission mechanism is connected to the computing unit; The mounting bracket is provided with three detection radars distributed in a triangle, the sensor sensing component is provided with an omnidirectional camera sensor, the omnidirectional camera sensor includes a plurality of wide-angle cameras arranged along a circumferential direction, and the interference signal transmitting device is arranged at the upper end of the omnidirectional camera sensor.

3. A radar-visual fusion method for an unmanned aerial vehicle intelligent capture system, characterized in that: The following steps are involved: S100, when an unauthorized target drone appears in a no-fly airspace, the acoustic array monitors the unique noise generated by the target drone during flight and transmits it back to the computing unit in the communication vehicle. After processing by the computing unit, the airspace azimuth angle where the target drone is located is obtained; S200, the vehicle-mounted UAV flies toward the airspace direction of the target UAV obtained in step S100, hovers near the target UAV, and uses detection radar and omnidirectional camera sensors to identify the target UAV; S300, combining the image obtained by the omnidirectional camera sensor after dedistortion, and identifying the target drone through the pre-trained neural network model deployed in the computing unit; S400, obtaining direction information of the target UAV through three-dimensional positioning, fusing the obtained direction information with the direction information obtained by the detection radar through a series of coordinate transformations, and obtaining position information of the target UAV; S500, causing the vehicle-mounted UAV to fly toward the target UAV, and using the rotating platform to control the jamming signal transmitter to adjust the direction toward the target UAV, so as to accurately capture the target UAV; S600: When the detection radar detects that the target UAV and the vehicle-mounted UAV are within the controllable range, the vehicle-mounted UAV uses the jamming signal transmitting device to transmit a sonic signal toward the target UAV to achieve the goal of shooting down the UAV.

4. The method for radar-visual fusion of a UAV intelligent capture system according to claim 3, characterized in that: In step S300, it includes: S310, calibrate the omnidirectional camera sensor to obtain the internal and external parameters and distortion matrix of the omnidirectional camera sensor; use the Shi-Tomasi algorithm to detect the corner points of black and white changes, where the detection function of the corner points is simplified to: In the formula, is the window function; is the energy function; To reflect the movement of the window; is the pixel coordinate; is the gray value function; is a real symmetric matrix; S320, according to the internal and external parameters of the omnidirectional camera sensor obtained in step S310, perform coordinate transformation operation on the panoramic image captured by the omnidirectional camera sensor, and use a rotation matrix and a translation vector to represent it. The rotation matrix is ​​responsible for the alignment of the coordinate axis direction, and the translation vector is responsible for the alignment of the origin. The center of the omnidirectional camera sensor is set as the origin, and the vertical omnidirectional camera sensor is pointed outward. Z Axis, establish a coordinate system, define it as the camera coordinate system, then the relationship between the camera coordinate system and the coordinate system of the actual space is as follows: In the formula, is the coordinate in the camera coordinate system; is the coordinate in the real space; R The matrix is ​​a rotation matrix, which is a unit orthogonal matrix containing the sine and cosine values ​​of the rotation angles in each direction; T is the translation transformation amount, i.e., the external parameter in step S310; The linear coordinates are expressed as: This forms the required camera coordinate system; S330, performing a dedistortion operation to establish a nonlinear distortion model: In the formula, is the internal reference obtained in step S310; is the distortion parameter; is the coordinate after dedistortion; r is the Euclidean distance between two points; The radial distortion is: ； S340, perform optical axis correction to make the optical axes of the camera array parallel after correction. According to the imaging principle of a single camera, correct the world coordinates of any point in the front and back space. I The corresponding image coordinates i The mapping relationships are: In the formula, c is the coordinate of the optical center in the world coordinate system; s is the projection of the distance from the object point to the optical center on the optical axis; A is the internal parameter matrix; R is the rotation matrix; the optical center remains unchanged before and after correction, that is c 1 and c 2 equal; After correction, the optical axis is parallel, which can be based on: Get the relationship between the optical center position before and after correction: ； S350, after the image array is calibrated, the size of the overlapping area is calculated according to the same edge corner point coordinates; the length of the overlapping area of ​​the images captured by camera 1 and camera 2 L h for: In the formula, is the length of the image captured by camera 1; is the edge corner point in the image captured by camera 1 The horizontal axis of Laser spot in the image captured by camera 2 The horizontal axis of The image stitching between camera 2 and camera 3, camera 3 and camera 4, and camera 4 and camera 5 is calculated using the same method.

5. The method for radar-visual fusion of a UAV intelligent capture system according to claim 3, characterized in that: In step S400, it includes: S410, transmitting the image information collected by the omnidirectional camera sensor to the computing unit through the optical fiber, and the computing unit identifies a series of related frame signals and line signals of SoF and EoL therein; S420, the video image obtained in step S410 is i Frame Set to I i , then the coordinates of the target UAV in the camera coordinate system are defined as [ X i , Y i , Z i ] T , the target detection model deployed on the computing unit obtains the pixel coordinates of the target drone as [ u i , v i ] T , coordinate transformation between two coordinate systems; S430, in step S420, the coordinates of the target UAV in the camera coordinate system are obtained [ X i , Y i , Z i ] T Then, use this coordinate to calculate the azimuth of the target drone θ and pitch angle 𝜙, to obtain the direction information of the target UAV in the actual space; S440, the detection radar communicates with the computing unit, obtains information on the target distance, azimuth, and pitch angle, and performs coordinate conversion; S450, the computing unit matches the collected radar sensor information with the coordinate information obtained by the method in step S420 of the image returned by the camera sensor, and obtains the difference frequency signal for detection using the information returned by the echo data of the detection radar.

6. The method for radar-visual fusion of unmanned aerial vehicle intelligent capture system according to claim 5, characterized in that: In step S440, if the visibility is low, before the signal is transmitted back to the detection radar, the background clutter power estimate is calculated by averaging the reference cells on both sides using the improved CA-CFAR algorithm, assuming that the target exists: In the formula, X i It is a protection unit. Y i is the reference unit, n is the total number of unilateral reference units; The detection threshold after obtaining the adjusted background power estimate is: in, α is the threshold factor, which determines the level of the detection threshold relative to the background power.

7. The method for radar-visual fusion of unmanned aerial vehicle intelligent capture system according to claim 5, characterized in that: In step S440, the detection radar acquires the data set and performs coordinate conversion for data comparison, converting the millimeter wave radar coordinates into world coordinates. P ( β , Q ) is in world coordinates: In the formula, the distance between the XOY planes of the two coordinate systems is Z 0, XOZ plane distance is Y w .

8. The method for radar-visual fusion of a UAV intelligent capture system according to claim 3, characterized in that: The detection radar sends a detection signal to the target UAV, and the target receives the signal and returns an echo signal to the millimeter wave radar. The transmission signal of the sawtooth frequency modulation continuous wave radar can be expressed as: In the formula, represents the initial phase, represents the radar signal amplitude, Indicates the starting frequency, It represents the frequency modulation slope, and its formula is as follows: In the formula, B Indicates the frequency bandwidth, T Indicates the frequency modulation repetition period; For a strong scattering point target with a distance of R and a velocity of v, its echo signal is expressed as follows: In the formula, C Indicates the amplitude attenuation of the signal; The sawtooth radar transmit signal is used as the local oscillator signal, and the difference frequency signal is formed by mixing the mixer and the echo. S B : ； From this, the frequency expression of the difference frequency signal is deduced: ； It can be concluded that: Center frequency ; FM slope ; FM Phase .

9. The method for radar-visual fusion of a UAV intelligent capture system according to claim 3, characterized in that: The detection radar adopts DBSCAN-CFSFDP clustering algorithm in data processing, and the local density information entropy formula is: In the formula, i Indicates the data set i data points, 1 b represents the logarithm with base 2, K Indicates that under a given neighborhood radius, i The ratio of the number of data points within the radius to the total number of points, that is i Probability distribution of local density of points; Then consider the distribution of the entire data set and average the local density information entropy of all data points, which is expressed as: In the formula, n is the number of data points in the dataset; Eps is the neighborhood radius, using the determined Eps Parameters, determine the noise points based on the number of data points in the neighborhood, remove the noise points, and obtain clean point cloud data; Select the starting point corresponding to the most stable information entropy change Eps , the density distribution between data points in the stable segment is relatively consistent; the Gini coefficient is used to evaluate the uneven distribution of data under different truncation distances and determine the optimal truncation distance best _ d c , and using the local density ρ i and the relative distance δ i The product of is used to make a descending graph of cluster center weights, and an adaptive exponential decay function curve is introduced to automatically determine the cluster center point; Local density: ； Clarity of CFSFDP algorithm clustering: In the formula, P i For the i The ratio of the cluster center weight of a point to the total cluster center weight is expressed as follows: In the formula, y i Indicates i The cluster center weight of each point is expressed as: 。 10. The method for radar-visual fusion of unmanned aerial vehicle intelligent capture system according to claim 3, characterized in that: When fusing radar and camera data, the decision-level PDA is used to fuse point cloud and image detection results: In the formula, S is the area of ​​the target detection box, i The number of pixels that the target detection box occupies in the image. The pixel value of each pixel will affect the size of the detection box. Project the 3D detection box generated by the radar onto the plane and treat it as a simplified rectangular box: In the formula, Y , Z Respectively represent the y-axis and z-axis coordinates of the detection box vertex on the two-dimensional plane; The basis for determining the overlap is set as the comparison value of the overlap area with the lidar detection frame and the camera detection frame area. The overlap can be used to accurately determine the degree of match between the lidar detection results and the camera detection results. The overlap formula is: In the formula, μ is the overlap of two detection frames, and s μ represents their overlapping area.

Images