Multi-source fusion detection and formation recognition system and method for unmanned aerial vehicle cluster

Abstract

The invention discloses a multi-source fusion detection and formation recognition system for an unmanned aerial vehicle cluster. The multi-source fusion detection and formation recognition system comprises a radar detection system, a photoelectric detection system and an information processing module. The invention further discloses a multi-source fusion detection and formation recognition method for the unmanned aerial vehicle cluster. The method comprises the steps that the radar detection system obtains the initial azimuth information of the unmanned aerial vehicle; the information processing module generates guidance information according to the initial azimuth information; the photoelectric detection system tracks the unmanned aerial vehicle according to the guidance information to obtain an image of the unmanned aerial vehicle; and the information processing module obtains the three-dimensional position information of the unmanned aerial vehicle according to the initial azimuth information and the unmanned aerial vehicle image, carries out the recognition of the unmanned aerial vehicle cluster formation pattern, and obtains a formation pattern structure. According to the method, accurate identification and three-dimensional coordinate acquisition of individual targets in a long-distance high-resolution unmanned aerial vehicle cluster and feature identification of the formation of the unmanned aerial vehicle can be realized, and high-reliability information support is provided for command, control and strike decisions.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle detection, and in particular relates to a multi-source fusion detection and individual formation recognition method and system for unmanned aerial vehicle clusters. Background Art

[0002] In order to achieve effective drone detection and identification, first of all, reliable detection capabilities at long distances, all day and in adverse weather conditions are the time guarantee for timely target discovery and uploading of targeting information. Secondly, large-field-of-view high-resolution image acquisition and comprehensive measurement of angles and distances are the technical support for accurate identification of individual drones and three-dimensional positioning. Finally, the identification of the formation of drone clusters is the source of information for the system to judge the intention of drones and formulate countermeasures. Therefore, single-dimensional sensors, simple individual identification or overall identification of formations cannot meet the detection requirements of anti-drone clusters.

[0003] The existing detection system uses laser imaging radar to collect drone information, which has weak performance in harsh environments, resulting in a lack of distance information for long-distance drone detection. In addition, the field of view and resolution of a single collected image are limited, and it does not have the ability to recognize formations. Optical components are used for video shooting, which cannot obtain the distance information of the drone and can only locate and track multiple target detection frames, lacking the ability to finely identify individual drones and formations. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned defects and provide a multi-source fusion detection and formation recognition system and method for drone swarms, which solves the technical problem that existing single-dimensional sensors, simple individual recognition or overall recognition of formations cannot meet the detection requirements of anti-drone swarms. The present invention can realize the accurate identification and three-dimensional coordinate acquisition of individual targets in drone swarms at long distances and high resolution, as well as the feature recognition of drone formations, and can provide long-distance warning information, high-precision targeting information and multi-dimensional situational awareness information for anti-drone swarm equipment, providing highly reliable information support for command and strike decisions.

[0005] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0006] The present invention proposes a multi-source fusion detection and individual formation recognition method and system for unmanned aerial vehicle clusters, which can improve the range, information dimension and detection accuracy of existing unmanned aerial vehicle cluster detection methods. The system includes a microwave radar, a television camera, an infrared thermal imager, a laser ranging module, an information processing module, etc. The microwave radar system is used to achieve large-scale scanning and long-distance detection of targets, provide preliminary position and orientation information to the system, and guide the optoelectronic system to perform scanning imaging. The television camera and the infrared thermal imager continuously shoot multiple frames of images of the target, splice them into a large-field high-resolution image through the improved algorithm of accelerated robust features (SURF), and input the visual background extraction (Vibe) neural network to identify individual targets of unmanned aerial vehicles. After the target information of visible, infrared and radar sensors is aligned in time and space, point trace correlation fusion is performed, and the radar distance information and optoelectronic high-precision angle information are comprehensively used to achieve three-dimensional positioning of the target. The optoelectronic imaging system guides the laser ranging module to perform single-point tracking and aiming measurement on targets that cannot be distinguished by microwave radar to supplement the target information. The three-dimensional coordinate information of multiple target points in the cluster is used to input the graph convolutional neural network for formation recognition. Therefore, through multi-source sensor fusion detection and phased intelligent processing algorithms, the precise identification of individual positions and reliable perception of the overall configuration in dense swarm scenarios of drones can be effectively achieved.

[0007] The present invention provides a multi-source fusion detection and formation recognition system for UAV clusters, including: a radar detection system, a photoelectric detection system and an information processing module;

[0008] The radar detection system is used to obtain the preliminary position information of the UAV and output the preliminary position information of the UAV to the information processing module; the preliminary position information includes distance information and rough angle information, the distance information facilitates the photoelectric imaging system to adjust to a suitable focal length, and the rough angle information facilitates the photoelectric imaging system to rotate to the corresponding position;

[0009] The information processing module generates guidance information according to the preliminary position information of the UAV input by the radar detection system, and outputs the guidance information to the photoelectric detection system;

[0010] The photoelectric detection system receives the guidance information input by the information processing module, tracks the UAV according to the guidance information, obtains the image of the UAV, and outputs the image of the UAV to the information processing module;

[0011] The information processing module obtains the three-dimensional position information of the UAV based on the preliminary orientation information input by the radar detection system and the UAV image input by the photoelectric detection system; the UAV cluster formation is identified based on the three-dimensional position information of the UAV to obtain the formation structure.

[0012] Furthermore, the radar detection system includes a microwave radar system and a radar azimuth turntable; the radar azimuth turntable drives the microwave radar system to conduct all-round target search with an azimuth coverage of 360° and an elevation coverage of 0 to 60°;

[0013] The photoelectric detection system includes a television camera, an infrared thermal imager, a pitch turntable and an optoelectronic azimuth turntable; the pitch turntable and the optoelectronic azimuth turntable control the pitch angle and azimuth angle of the television camera and the infrared thermal imager according to the guidance information. The television camera is used to obtain daytime UAV images, and the infrared thermal imager is used to obtain all-day UAV images.

[0014] Furthermore, the UAV image acquired by the photoelectric detection system is a multi-frame continuous acquisition image; the preliminary orientation information includes the distance information and rough angle information of the UAV;

[0015] The method for the information processing module to obtain the three-dimensional position information of the UAV according to the preliminary orientation information input by the radar detection system and the UAV image input by the photoelectric detection system includes:

[0016] By accelerating the robust feature improvement algorithm, the multiple frames of continuously collected images obtained by the photoelectric detection system are stitched together to obtain a stitched image; the stitched image is input into the visual background extraction neural network to identify the individual targets of the UAV and obtain the precise angular information of the UAV;

[0017] After the preliminary orientation information and the precise angle information of the UAV are aligned in time and space, point-trace correlation fusion processing is performed to obtain the three-dimensional position information of the UAV.

[0018] Furthermore, the above-mentioned multi-source fusion detection and formation recognition system for drone clusters also includes a laser ranging module;

[0019] The laser ranging module performs single-point ranging on the UAV that fails the point-track association fusion processing according to the UAV's precise angular information to obtain the UAV's distance information; the information processing module aligns the UAV's distance information obtained by the laser ranging module with the UAV's precise angular information in time and space, and then performs point-track association fusion processing to obtain the UAV's three-dimensional position information.

[0020] A multi-source fusion detection and formation recognition method for UAV clusters, comprising:

[0021] The S1 radar detection system obtains the preliminary position information of the UAV and outputs the preliminary position information of the UAV to the information processing module;

[0022] The S2 information processing module generates guidance information based on the preliminary position information of the UAV input by the radar detection system, and outputs the guidance information to the photoelectric detection system;

[0023] The S3 photoelectric detection system receives the guidance information input by the information processing module, tracks the UAV according to the guidance information, obtains the image of the UAV, and outputs the image of the UAV to the information processing module;

[0024] The S4 information processing module obtains the three-dimensional position information of the UAV based on the preliminary orientation information input by the radar detection system and the UAV image input by the optoelectronic detection system; it identifies the formation of the UAV cluster based on the three-dimensional position information of the UAV and obtains the formation structure.

[0025] Furthermore, the UAV image acquired by the photoelectric detection system is a multi-frame continuous acquisition image; the preliminary orientation information includes the distance information and rough angle information of the UAV;

[0026] In step S4, the information processing module obtains the three-dimensional position information of the drone according to the preliminary orientation information input by the radar detection system and the drone image input by the photoelectric detection system. The method includes:

[0027] S4.1 uses an accelerated robust feature improvement algorithm to stitch together multiple frames of continuously acquired images acquired by the photoelectric detection system to obtain a stitched image; the stitched image is input into a visual background extraction neural network to identify individual drone targets and obtain accurate angular information of the drone;

[0028] S4.2 performs point-track correlation fusion processing after aligning the preliminary orientation information and the precise angle information of the UAV in time and space, and obtains the three-dimensional position information of the UAV.

[0029] Furthermore, the above-mentioned multi-source fusion detection and formation recognition method for drone clusters also includes:

[0030] According to the precise angular information of the UAV, the laser ranging module is used to perform single-point ranging on the UAV that fails the point-track association fusion processing to obtain the distance information of the UAV; the information processing module aligns the distance information of the UAV obtained by the laser ranging module with the precise angular information of the UAV in time and space, and then performs point-track association fusion processing to obtain the three-dimensional position information of the UAV.

[0031] Furthermore, the method of step S4.1 includes:

[0032] S4.1.1 preprocessing the multiple frames of continuously acquired images, wherein the preprocessing includes format normalization, digital filtering or target enhancement;

[0033] S4.1.2 extract feature points from each frame of image according to the accelerated segmentation detection feature algorithm to obtain feature points of each frame of image;

[0034] S4.1.3 describes the feature points of each frame image using an accelerated robust feature algorithm to obtain a description vector of the feature points of each frame image;

[0035] S4.1.4 Based on the description vector of the feature points of each frame of image, the feature points are matched using the kd-tree-based nearest neighbor search algorithm to determine the feature points that match each other between two adjacent frames of image, that is, to find the repeated parts between two adjacent frames of image;

[0036] S4.1.5 obtains the change matrix between the feature points that match each other in two adjacent frames of images based on the random sampling consistency algorithm;

[0037] S4.1.6 Based on a weighted fusion algorithm with fade-in and fade-out, the change matrix is ​​used to stitch multiple frames of continuously collected images to obtain a stitched image;

[0038] S4.1.7 Perform background modeling on the first frame spliced ​​image based on the neighborhood sampling method;

[0039] S4.1.8 Based on the Euclidean distance discrimination method, classify the foreground object and background of the i-th frame of the spliced ​​image to obtain the background points and foreground objects of the spliced ​​image; i≥1;

[0040] S4.1.9 Based on the background points and foreground targets of the i-th frame stitched image, a random update method is used to update the background model of the i+1-th frame stitched image; specifically, assuming that the background points of the i-th frame have been updated to the background points determined by the i-1-th frame, if the i+1-th frame is updated, then the background points are updated to the newly determined background points of the i-th frame; if the i+1-th frame is not updated, then the original background points of the i-th frame are maintained, that is, the background points determined by the i-1-th frame;

[0041] S4.1.10 Repeat steps S4.1.8 and S4.1.9 to output the foreground target in each frame of the stitched image, and then use triangulation formulas and other calculation methods to obtain the precise angular information of the UAV.

[0042] Further, the method of step S4.2 includes:

[0043] S4.2.1 Based on the command and control center time, synchronize the preliminary azimuth information and the precise angle information of the UAV to the standard time; using the geocentric coordinate system as the reference coordinate system for spatial alignment, convert the local polar coordinates of the target relative to the radar detection system to the local polar coordinates of the target relative to the photoelectric detection system;

[0044] S4.2.2 Perform point-track association fusion processing based on the rough angle information in the preliminary orientation information and the precise angle information of the UAV;

[0045] S4.2.3 The preliminary orientation information and precise angle information of the UAV that are successfully processed by point-track association fusion are compressed and fused through dimensional expansion and lossless transformation of the fusion measurement equation to obtain the three-dimensional position information of the UAV.

[0046] Furthermore, in step S4, the method of identifying the formation of the drone cluster according to the three-dimensional position information of the drone to obtain the formation structure includes:

[0047] S4.3.1 Use the three-dimensional position information of each drone as input data to model the drone cluster graph;

[0048] S4.3.2 extract features from the drone cluster graph; the extracted features include connection features, motion features, and node features of the drone cluster graph;

[0049] S4.3.3 Based on the extracted features, classification and recognition are performed through a graph convolutional neural network, and the formation of the drone cluster after classification and recognition is output.

[0050] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0051] (1) The present invention comprehensively utilizes the advantages of microwave radar, such as long detection distance, large scanning range, little interference from adverse weather conditions, providing distance information, and being able to work all day long; the advantages of visible-infrared radar, such as high resolution, rich detail information, and accurate azimuth information; and the advantages of laser ranging, such as high single-point angle and distance measurement accuracy. The present invention achieves the acquisition of target three-dimensional coordinate information through the spatiotemporal alignment of heterogeneous information and point trace association, as well as single-point measurement under multi-layer guidance information. The present invention has the advantages of long positioning distance, high positioning accuracy, and comprehensive information.

[0052] (2) The present invention uses a multi-frame spliced ​​large-field-of-view high-resolution image as input, and realizes the recognition of individual drone targets through an intelligent target detection algorithm based on foreground segmentation. It has the advantages of a large detection field of view, high target recognition accuracy, and resistance to vibration interference;

[0053] (3) The present invention uses the three-dimensional coordinates of multiple targets as input and realizes the recognition of cluster formations through graph neural networks, which has the advantages of comprehensive information, clear nodes and good visualization effects;

[0054] (4) The present invention provides the command and control system with high-precision target information and multi-dimensional situation information through different forms of neural network intelligent algorithms, and has the advantages of high degree of automation, high reliability and comprehensive information. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of a multi-source fusion detection and formation recognition system for UAV clusters of the present invention;

[0056] Figure 2 It is a flow chart of the multi-source fusion detection and formation recognition method for UAV clusters of the present invention;

[0057] Figure 3 It is a schematic diagram of the algorithm flow of multi-frame image stitching and drone individual target recognition of the present invention;

[0058] Figure 4 It is a flow chart of a target three-dimensional positioning algorithm based on multi-source information fusion of visible-infrared, radar and laser ranging according to the present invention;

[0059] Figure 5 The figure is a schematic diagram of the algorithm flow for the UAV cluster formation recognition of the present invention. DETAILED DESCRIPTION

[0060] The following detailed description of the present invention will make the features and advantages of the present invention more clear and explicit.

[0061] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0062] The present invention provides a multi-source fusion detection and individual formation recognition method for UAV clusters, comprising:

[0063] The microwave radar can search for targets in all directions at 360 degrees, obtain the preliminary position of the long-distance drone cluster targets, and guide the photoelectric detection system to perform fine scanning and imaging;

[0064] The television camera and infrared thermal imager are controlled by the servo system to perform two-dimensional scanning in pitch and azimuth, and take array-type multi-frame continuous images of the target;

[0065] Information processing for image stitching and target recognition: Each group of multi-frame images is stitched based on the Speed ​​Up Robust Features (SURF) intelligent algorithm to form a large-field-of-view high-resolution image that is input into the Visual Background Extraction (Vibe) neural network to identify the type of moving targets and obtain angle information.

[0066] Perform fusion positioning processing based on multi-source information. After aligning the time and space of microwave radar, TV camera and infrared thermal imager data, perform correlation processing on target points, and use the fused radar distance information and photoelectric angle information to obtain the target position. At the same time, the angle information of the photoelectric imaging system guides the laser ranging module to perform single-point tracking and aiming measurement on the target that fails to be correlated, so as to achieve three-dimensional positioning of all target points.

[0067] Perform formation recognition based on multi-point coordinates. Use the three-dimensional positioning of each target point to form a directed modeling graph, input the graph convolutional neural network to recognize the cluster formation, and use the display module to realize the visualization of individual drones and formations.

[0068] The present invention can realize the effective recognition of mobile small targets in complex environments, expand the field of view, and reduce the impact of platform vibration. The information processing method for image splicing and target recognition of the present invention includes:

[0069] Feature points of multiple frames in each group are extracted, described and matched, and the transformation matrix between adjacent images is solved. They are fused and spliced ​​based on a weighted fusion algorithm with fade-in and fade-out to output a large-field-of-view high-resolution image.

[0070] The background model of the first frame of the stitched image is modeled based on the neighborhood sampling method, and the background model of each frame of the stitched image is updated by a random update method with a certain probability according to the scene characteristics, so as to reduce the target recognition error caused by too frequent background updates due to platform vibration.

[0071] The foreground target and modeling background are classified for the first frame image and subsequent updated images according to the Euclidean distance discrimination method, and the foreground moving target of the micro-UAV in each frame image is output.

[0072] The present invention can make the advantages and disadvantages of multiple sensors complement each other and obtain comprehensive and high-precision three-dimensional coordinate information of the target all day long. The method of fusion positioning processing based on multi-source information includes:

[0073] The interpolation and extrapolation methods are used to synchronize each perception platform to the standard time of the command and control center. At the same time, the geocentric coordinate system is used as the reference coordinate system to convert all target position data into local polar coordinates relative to the infrared sensor.

[0074] Visible and infrared sensors are used to obtain the target direction angle through image recognition, and the target direction angle solved by the radar echo signal is associated with the target point trace based on the angle dimension. The association decision threshold is determined by the significance level, and the associated point trace is determined.

[0075] The successfully associated radar, visible and infrared data are compressed and fused through dimensional expansion and lossless transformation of the fusion measurement equation, and the radar distance information and photoelectric sensor angle information of the target point are comprehensively utilized to achieve high-precision three-dimensional coordinate acquisition of the target.

[0076] For target points that fail to be associated due to indistinguishability within the radar angular range, the angle information obtained by the visible and infrared sensors guides the laser ranging module to measure point targets and obtain the three-dimensional position coordinates of all targets.

[0077] The present invention can realize the intelligent recognition and visual representation of the irregular data structure of the drone cluster. The method for performing formation recognition processing based on multi-point coordinates includes:

[0078] The three-dimensional position coordinates of multiple target points in the cluster are modeled to form a directed graph set. Feature extraction is performed based on connection features, motion features and node features. The graph structure data is converted into Euclidean structure data through a graph convolutional neural network to achieve effective recognition and description of cluster formations.

[0079] The present invention provides a multi-source fusion detection and individual formation recognition system for UAV clusters, including a microwave radar, a television camera, an infrared thermal imager, a laser ranging module, an information processing module, a display module, a servo turntable (the servo turntable includes Figure 1 Central system azimuth turntable, radar azimuth turntable, optoelectronic azimuth turntable and optoelectronic pitch turntable) and power supply module:

[0080] Microwave radar system, used to achieve large-range scanning and long-distance detection of targets, includes a transmitter, a transmitting antenna, a receiver, a receiving antenna, a signal processor and a display.

[0081] The television camera is used for fine imaging of drone cluster targets in the visible light band during the day, including a visible optical imaging component, a CMOS detector component and an optical controller, which can achieve a 5-speed zoom of the visible light imaging lens.

[0082] The infrared thermal imager is used for fine imaging of drone cluster targets in the infrared band at all times of the day. It includes infrared optical imaging components, infrared focal plane detection components and infrared electronics components. It can achieve a 4-speed zoom of the mid-infrared imaging lens, and the infrared and visible light have good optical axis consistency.

[0083] The laser ranging module is used for high-precision single-point ranging of UAVs in the infrared band all day long. It includes a laser optical system, a laser transmitter, a laser receiver and a laser information processor. It can measure the distance of the target on the basis of stable tracking of the target by the optoelectronic system, obtain its distance information in real time, and has good optical axis consistency with the optoelectronic imaging sensor.

[0084] The information processing module is used for the fusion processing of radar, visible light and infrared data, and uses deep learning neural networks to realize the recognition of moving targets and formation structures.

[0085] The display module is used to provide a visualization interface of the three-dimensional position coordinates of multiple target points in the cluster and the formation, and is connected to the information processing module.

[0086] The servo turntable is used for fixed installation of microwave radar system, television camera, infrared thermal imager and information processing module, and is used for realizing all-round rotation of microwave radar system and two-dimensional rotation of pitch and azimuth of television camera and infrared thermal imager.

[0087] The power supply module is used to supply power to microwave radar, television camera, infrared thermal imager, information processing module, display module and servo turntable.

[0088] The present invention comprehensively utilizes the advantages of microwave radar, such as long detection distance, large scanning range, little interference from adverse weather conditions, the ability to provide distance information, and the ability to work around the clock; the advantages of visible-infrared high resolution, rich detail information, and accurate azimuth information; and the advantages of laser ranging, which have high accuracy in single-point angle and distance measurement. The present invention acquires the three-dimensional coordinate information of the target through the spatiotemporal alignment and point trace association of heterogeneous information and single-point measurement under multi-layer guidance information. It has the advantages of long positioning distance, high positioning accuracy, and comprehensive information.

[0089] The present invention uses multi-frame spliced ​​large-field-of-view high-resolution images as input, and realizes the recognition of individual drone targets through an intelligent target detection algorithm based on foreground segmentation, which has the advantages of a large detection field of view, high target recognition accuracy and resistance to vibration interference; uses the three-dimensional coordinates of multiple targets as input, and realizes the recognition of cluster formations through a graph neural network, which has the advantages of comprehensive information, clear nodes and good visualization effects.

[0090] The present invention provides the command and control system with high-precision target information and multi-dimensional situation information through different forms of neural network intelligent algorithms, and has the advantages of high degree of automation, high reliability and comprehensive information.

[0091] Example:

[0092] The present invention realizes large-scale scanning and long-distance detection of targets through a microwave radar system, provides preliminary position and azimuth information (i.e. preliminary azimuth, which includes distance information and rough angle information) to the system, and guides the photoelectric system to perform scanning imaging. The microwave radar system adopts a fully solid-state, fully coherent, pulse compression technology system, the antenna adopts a flat waveguide slot array antenna, the elevation direction adopts one-dimensional phase scanning, and the azimuth direction realizes mechanical scanning through a servo turntable, so as to achieve azimuth and elevation coverage of the monitoring area.

[0093] The optoelectronic system integrates a television camera and a medium-wave infrared thermal imager, and realizes continuous shooting of multiple frames of images through two-dimensional scanning. It is characterized in that the television camera includes a visible optical imaging component, a CMOS detector component and an optical controller; the medium-wave infrared thermal imager includes a continuous zoom infrared optical imaging component, an infrared focal plane detection component and an infrared electronics component.

[0094] Multiple frames of images taken continuously are stitched into a large-field-of-view high-resolution image through an image stitching algorithm, which is input into a deep learning neural network for individual target recognition of the drone. Image stitching technology based on the improved speeded up robust features (SURF) algorithm is used to stitch 5×5 continuous small-field-of-view images.

[0095] Based on the individual target recognition information of the photoelectric image, the input signal processing unit is fused with the microwave radar information, and the visual background extraction (Vibe) target detection algorithm is used to detect individual UAV targets, and the target azimuth information based on the optical and infrared images is obtained.

[0096] Visible, infrared and radar information are fused and processed to align in time and space, realizing track association based on heterogeneous information and fusion positioning of multi-source data, including: aligning the clocks of multivariate data information by interpolation and extrapolation based on the command and control center time; and realizing spatial alignment of visible, infrared and radar information by using the geocentric coordinate system (ECEF) as the medium.

[0097] Heterogeneous information track association and target fusion positioning are used to obtain the three-dimensional coordinate information of multiple target points in the cluster for formation recognition, including target point track association based on the significance level threshold between visible and infrared angle information and radar angle information. After fusion processing, high-precision three-dimensional coordinate acquisition of associated targets is achieved using radar distance information and visible and infrared angle information.

[0098] The angle information obtained by the visible and infrared sensors guides the laser ranging module to perform single-point ranging on target points that fail to be associated due to indistinguishable radar direction angles, and supplement the three-dimensional position coordinates of all targets in the cluster.

[0099] Formation recognition is achieved by inputting topological structures into a deep learning network, including: using a graph convolutional network to perform deep learning classification from three dimensions: connection features, motion features, and node features of graph nodes to identify cluster formations.

[0100] Embodiment 1:

[0101] A specific embodiment of the present invention, as Figure 1 , discloses a multi-source detection system for UAV clusters, including: a servo turntable, a radar detection system, an optoelectronic detection system, an information processing module, a display module and a power supply module.

[0102] The radar detection system, photoelectric detection system and information processing module are arranged on the system azimuth turntable.

[0103] The radar detection system includes a microwave radar system controlled by a radar azimuth turntable for scanning, and the photoelectric detection system includes a television camera, an infrared thermal imager and a laser ranging module controlled by a photoelectric azimuth turntable and a pitch turntable for two-dimensional scanning.

[0104] The microwave radar system is used for all-day detection, positioning and tracking of dense swarms of drones, and provides azimuth guidance information to the optoelectronic detection system. The microwave radar system includes a transmitter, a transmitting antenna, a receiver, a receiving antenna, a signal processor and a display.

[0105] The television camera is used for high-definition monitoring and high-precision tracking imaging of daytime targets in the visible light band, and provides azimuth guidance information to the laser ranging module. The television camera includes an optical imaging component, a CMOS detector component with an array size of 3840×2160, and an optical controller. The optical controller realizes functions such as optical system zoom, dimming and image preprocessing, and is responsible for the adaptation of external electrical and data interfaces.

[0106] The infrared thermal imager is used for high-definition monitoring and high-precision tracking imaging of targets in the infrared band all day long, and provides azimuth guidance information to the laser ranging module. The infrared thermal imager includes a continuous zoom infrared optical imaging component, an infrared focal plane detection component with an array size of 1280×1024, and an infrared electronic component. The electronic component completes the drive control of the focal plane device, image signal acquisition, non-uniformity correction and image detail enhancement processing, and is responsible for the adaptation of external electrical and data interfaces.

[0107] The laser ranging module is used for high-precision tracking and ranging of targets in the infrared band all day long. The laser ranging module includes a laser optical system, a laser transmitter, a laser receiver and a laser information processor. It can measure the target based on the pulse time ranging method on the basis of stable target tracking by the optoelectronic system.

[0108] The information processing module is used for information processing such as spatiotemporal alignment, point-trace association and fusion positioning of radar, visible light and infrared data. It uses deep learning algorithms such as visual background extraction (Vibe) and graph convolutional neural network to realize the recognition of moving targets and formation structures.

[0109] The display module is connected to the information processing module and is used to provide a visualization interface of the three-dimensional position coordinates of multiple UAV targets in the cluster and the cluster formation.

[0110] The power supply module is used to supply power to the information processing module, the display module, the servo turntable, the radar detection system and the photoelectric detection system.

[0111] Embodiment 2:

[0112] Another specific embodiment of the present invention is as follows Figure 2, discloses a multi-source detection and intelligent identification method for drone clusters, including the following steps:

[0113] Step 1: The microwave radar conducts an all-round target search with an azimuth coverage of 0 to 360 degrees and an elevation coverage of 0 to 60 degrees. It uses a combination of pulse Doppler processing technology and long-term echo accumulation technology to locate the target, provide preliminary azimuth information (preliminary azimuth information includes distance information and rough angle information), and guide the photoelectric detection system to perform scanning imaging.

[0114] Step 2: Use the servo turntable to control the azimuth and pitch angles of the TV camera and infrared thermal imager to perform two-dimensional scanning, continuously shoot 25 visible light and infrared images with an array size of 5×5 at a frame rate of 100 Hz, and adjust the focal length of the optical system of the TV camera and infrared thermal imager so that the entire drone cluster is within the field of view of the 25 images.

[0115] Step 3: stitch the 25 consecutively shot frames into a high-resolution image with a large field of view, and use the neural network deep learning algorithm to identify individual drone targets, and obtain the azimuth and pitch angle information (i.e., precise angle information) of the target relative to the TV camera and infrared thermal imager. The main steps of the deep learning algorithm include: background modeling, background foreground separation, and background update;

[0116] Step 4: Align the information collected by the television camera, infrared thermal imager and microwave radar in time and space, perform point-trace correlation fusion processing, and then comprehensively utilize the precise angle information obtained by the photoelectric detection system and the distance information obtained by the radar detection system to obtain the high-precision three-dimensional position coordinates of the target.

[0117] Step 5: For target points that fail to be associated due to indistinguishability within the radar angular range, the angle information obtained by the visible and infrared sensors guides the laser ranging module to measure the point targets, thereby supplementing the three-dimensional position coordinates of all targets.

[0118] Step six, construct the three-dimensional coordinate information of multiple targets in the drone cluster into a directed modeling graph, use the neural network deep learning algorithm to identify the drone cluster formation, and obtain the formation structure.

[0119] In step 3, the algorithm flow of multi-frame image stitching and drone individual target recognition is as follows: Figure 3 , including the following steps:

[0120] Step (1) performs image preprocessing including format normalization, digital filtering, and target enhancement on multiple frames of continuously acquired images.

[0121] Step (2) extracts feature points using the Fast Segmentation Detection (FAST) algorithm. x∈c |I(x)-I(P)|>ε d To determine the absolute difference N between the grayscale of each pixel I(x) and the grayscale of the surrounding 16 pixels I(P), the value exceeds the threshold ε d Then the point is judged as a corner point and used as a feature point in the image. In the formula, c represents the set of positions of the surrounding 16 pixels, and x represents the position of the current calculated pixel.

[0122] Step (3) describes the feature points based on the SURF algorithm. Calculate the feature descriptor in the rectangle centered on the feature point and divide it into 16 subintervals. Calculate the response value d in the x and y directions of the Haar wavelet feature in each subinterval. x and d y , as the description vector of the feature point V = (∑d x ,∑|d x |,∑d y ,∑|d y |).

[0123] Step (4) performs feature point matching based on the kd-tree nearest neighbor search (BBF) algorithm. Calculate the distance l of all feature points between adjacent frame images. When the ratio of the closest feature point to the second closest point is less than the threshold T, , the two points are considered to be matching feature points.

[0124] Step (5), based on the mutually matched feature points, the transformation matrix between the mutually matched feature points is obtained, specifically, the transformation matrix is ​​solved based on the random sampling consensus (RANSAC) algorithm. The singular value decomposition method is used to calculate the transformation matrix between the images. (x, y) and (x', y') are the feature point positions of adjacent frame images to be stitched, and s is the scale parameter.

[0125] Step (6), image fusion and stitching are performed based on a weighted fusion algorithm with gradual in and out. The stitched large field of view and high resolution image is:

[0126]

[0127] Where a and b are the horizontal and vertical coordinates of the pixel points in the image, x1 and x2 are the images to be stitched, X is the stitched image, γ is the weighting factor, γ = wd / w, w is the horizontal coordinate distance of the overlapping part of the stitched image, and wd is the horizontal coordinate distance from the pixel point of the overlapping part of the stitched image to the overlapping part.

[0128] Step (7), based on the neighborhood sampling method, the background modeling is performed on the first frame spliced ​​image, that is, the background sample set is obtained according to the spliced ​​image. The background sample set M(x) = {v1, v2, v3, ..., v n} is the pixel value v(x) of a random coordinate point in the 3×3 neighborhood of each coordinate point in the first frame image.

[0129] Step (8) classifies the foreground target and the background based on the Euclidean distance. Map the pixel point set to the two-dimensional Euclidean space, take the pixel value v(x) at the pixel point x as the center of the circle, and the threshold R as the radius to construct a circular area SR(v(x)). If the Euclidean distance d(v1,v2) between the pixel value at the pixel point x and the sample value in the sample set is less than R, that is, within the circular area, there is a strong similarity, then the pixel point is judged to belong to the background point, otherwise it is a foreground target. In this step, based on the background sample set, the foreground target and the newly judged background are obtained.

[0130] Step (9) uses a random update method to update the background model. When a certain v(x) pixel is judged as a background point, the sample value in the background sample set is updated with a probability of 1 / α, where α is a time sampling parameter determined by the characteristics of the scene in the image, thereby better reducing the impact of image vibration caused by sensor shaking.

[0131] Step (10), repeat the background model update and foreground background classification judgment, and output the drone foreground target in each frame of the image.

[0132] In step 4, the target three-dimensional positioning algorithm flow based on single-point ranging under visible-infrared, radar multi-source information fusion and multi-layer guidance information is as follows: Figure 4 , including the following steps:

[0133] Step (1) aligns the multi-source sensor information of the television camera (visible light sensor), infrared thermal imager and microwave radar in time and space. Based on the command center time, the classical time alignment method of interpolation and extrapolation is used to synchronize each perception platform to the standard time. At the same time, the geocentric coordinate system is used as the reference coordinate system for spatial alignment, and the local polar coordinates of the target relative to the radar system are converted to the local polar coordinates relative to the infrared sensor.

[0134] Step (2) is to associate the radar with the photoelectric target point traces based on the angle information. The target image recognition result of the visible-infrared sensor is associated with the target point trace reported by the radar based on the angle dimension, and the association decision threshold is determined by the significance level to determine the associated point trace. In this step, the target point that is successfully associated is obtained based on the precise angle information of the visible-infrared sensor and the rough angle information of the radar.

[0135] Step (3) uses the successfully associated radar, visible, and infrared data inputs to fuse the target points. By expanding the dimensionality of the fusion measurement equation and performing lossless transformation, information compression fusion is performed when three or two sensors are input simultaneously, as well as high-precision approximation of nonlinear measurement statistical characteristics. Then, lossless Kalman time-domain filtering is used to achieve high-precision estimation of the target state. In this step, based on the successfully associated target points, the fusion information of the successfully associated target is obtained.

[0136] Step (4) uses the multi-source information after the target point trace is associated and fused for precise positioning. Based on the radar distance information of the associated target point and the angle information of the visible light and infrared sensors, multi-source fusion is used to obtain the high-precision three-dimensional coordinates of the target.

[0137] Step (5) uses laser ranging to accurately locate single-point targets. Based on the angle information of visible light and infrared sensors, the laser ranging module is guided to perform single-point tracking and ranging for the remaining target points that failed to be associated, thus achieving complete three-dimensional positioning of all targets in the cluster.

[0138] In step six, the UAV cluster formation recognition algorithm process is as follows: Figure 5 , including the following steps:

[0139] Step (1) is to model the drone cluster graph. The three-dimensional position coordinates of multiple target points in the cluster are used as input data to form a directed graph G = {V, E, R} to represent a drone swarm network, where the non-empty set V = {1, 2, ..., v i ,…,N} represents the set of nodes, N is the number of nodes; E={e i,j =(i,j)|i,j∈V} represents the set of edges, e i,j Indicates that there is an edge connection between nodes i and j; R represents the node set of the edge, r i,j Represents the edge from node i to node j.

[0140] Step (2) extracts features of the modeling graph. Connection features, motion features, and node features are expressed through nodes, edges, and the relationship between edges and nodes, and the descriptive features of the graph are extracted based on the connection features, motion features, and node features.

[0141] Step (3) is to perform classification and recognition through graph convolutional neural network. Graph neural network can convert graph structure data into Euclidean structure data, so that the formation of drone clusters can be modeled and described, so as to perform network training and other operations. With the directed modeling graph as input, its pattern features are extracted through graph convolution operation.

[0142] Step (4), output the drone cluster formation after neural network classification.

[0143] In summary, the multi-source fusion detection and individual formation recognition method and system for drone swarms proposed in the present invention can achieve timely detection of targets at long distances, obtain high-precision complete three-dimensional coordinate information of targets, and generate a visualized formation to provide overall situation information of the swarm. In view of the excellent recognition characteristics of the proposed detection system and fusion intelligent processing algorithm, it can be applied to anti-drone swarm equipment, providing long-range warning information, high-precision target information and multi-dimensional situation awareness information for the equipment, and providing highly reliable information support for command and strike decisions.

[0144] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.

[0145] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A multi-source fusion detection and formation recognition system for drone clusters, characterized by: include: Radar detection systems, optoelectronic detection systems and information processing modules; The radar detection system is used to obtain the preliminary position information of the UAV and output the preliminary position information of the UAV to the information processing module; The information processing module generates guidance information according to the preliminary position information of the UAV input by the radar detection system, and outputs the guidance information to the photoelectric detection system; The photoelectric detection system receives the guidance information input by the information processing module, tracks the UAV according to the guidance information, obtains the image of the UAV, and outputs the image of the UAV to the information processing module; The information processing module obtains the three-dimensional position information of the UAV based on the preliminary orientation information input by the radar detection system and the UAV image input by the photoelectric detection system; the UAV cluster formation is identified based on the three-dimensional position information of the UAV to obtain the formation structure.

2. The multi-source fusion detection and formation recognition system for UAV clusters according to claim 1 is characterized in that: The radar detection system includes a microwave radar system and a radar azimuth turntable; The radar azimuth turntable drives the microwave radar system to conduct all-round target search with azimuth coverage of 360° and elevation coverage of 0-60°; The photoelectric detection system includes a television camera, an infrared thermal imager, a pitch turntable and an optoelectronic azimuth turntable; the pitch turntable and the optoelectronic azimuth turntable control the pitch angle and azimuth angle of the television camera and the infrared thermal imager according to the guidance information. The television camera is used to obtain daytime UAV images, and the infrared thermal imager is used to obtain all-day UAV images.

3. The multi-source fusion detection and formation recognition system for UAV clusters according to claim 1 is characterized in that: The UAV images acquired by the photoelectric detection system are multi-frame continuous acquisition images; the preliminary orientation information includes the distance information and rough angle information of the UAV; The method for the information processing module to obtain the three-dimensional position information of the UAV according to the preliminary orientation information input by the radar detection system and the UAV image input by the photoelectric detection system includes: By accelerating the robust feature improvement algorithm, the multiple frames of continuously collected images obtained by the photoelectric detection system are stitched together to obtain a stitched image; the stitched image is input into the visual background extraction neural network to identify the individual targets of the UAV and obtain the precise angular information of the UAV; After the preliminary orientation information and the precise angle information of the UAV are aligned in time and space, point-trace correlation fusion processing is performed to obtain the three-dimensional position information of the UAV.

4. The multi-source fusion detection and formation recognition system for UAV clusters according to claim 3 is characterized in that: Also includes a laser ranging module; The laser ranging module performs single-point ranging on the UAV that fails the point-track association fusion processing according to the UAV's precise angular information to obtain the UAV's distance information; the information processing module aligns the UAV's distance information obtained by the laser ranging module with the UAV's precise angular information in time and space, and then performs point-track association fusion processing to obtain the UAV's three-dimensional position information.

5. A multi-source fusion detection and formation recognition method for drone clusters, characterized in that: include: The S1 radar detection system obtains the preliminary position information of the UAV and outputs the preliminary position information of the UAV to the information processing module; The S2 information processing module generates guidance information based on the preliminary position information of the UAV input by the radar detection system, and outputs the guidance information to the photoelectric detection system; The S3 photoelectric detection system receives the guidance information input by the information processing module, tracks the UAV according to the guidance information, obtains the image of the UAV, and outputs the image of the UAV to the information processing module; The S4 information processing module obtains the three-dimensional position information of the UAV based on the preliminary orientation information input by the radar detection system and the UAV image input by the optoelectronic detection system; it identifies the formation of the UAV cluster based on the three-dimensional position information of the UAV and obtains the formation structure.

6. The multi-source fusion detection and formation recognition method for UAV clusters according to claim 5 is characterized in that: The UAV images acquired by the photoelectric detection system are multi-frame continuous acquisition images; the preliminary orientation information includes the distance information and rough angle information of the UAV; In step S4, the information processing module obtains the three-dimensional position information of the drone according to the preliminary orientation information input by the radar detection system and the drone image input by the photoelectric detection system. The method includes: S4.1 uses an accelerated robust feature improvement algorithm to stitch together multiple frames of continuously acquired images acquired by the photoelectric detection system to obtain a stitched image; the stitched image is input into a visual background extraction neural network to identify individual drone targets and obtain accurate angular information of the drone; S4.2 performs point-track correlation fusion processing after aligning the preliminary orientation information and the precise angle information of the UAV in time and space, and obtains the three-dimensional position information of the UAV.

7. The multi-source fusion detection and formation recognition method for UAV clusters according to claim 5 is characterized in that: Also includes: According to the precise angle information of the UAV, the laser ranging module is used to perform single-point ranging on the UAV that fails the point-track association fusion processing to obtain the distance information of the UAV; The information processing module aligns the distance information of the UAV obtained by the laser ranging module and the precise angle information of the UAV in time and space, and then performs point-trace correlation fusion processing to obtain the three-dimensional position information of the UAV.

8. The multi-source fusion detection and formation recognition system for UAV clusters according to claim 6 is characterized in that: The method of step S4.1 includes: S4.1.1 preprocessing the multiple frames of continuously acquired images, wherein the preprocessing includes format normalization, digital filtering or target enhancement; S4.1.2 extract feature points from each frame of image according to the accelerated segmentation detection feature algorithm to obtain feature points of each frame of image; S4.1.3 describes the feature points of each frame image using an accelerated robust feature algorithm to obtain a description vector of the feature points of each frame image; S4.1.4 Based on the description vector of the feature points of each frame of image, a kd-tree-based nearest neighbor search algorithm is used to match the feature points to determine the matching feature points of two adjacent frames of image; S4.1.5 obtains the change matrix between the feature points that match each other in two adjacent frames of images based on the random sampling consistency algorithm; S4.1.6 Based on a weighted fusion algorithm with fade-in and fade-out, the change matrix is ​​used to stitch multiple frames of continuously collected images to obtain a stitched image; S4.1.7 Perform background modeling on the first frame spliced ​​image based on the neighborhood sampling method; S4.1.8 Based on the Euclidean distance discrimination method, classify the foreground object and background of the i-th frame of the spliced ​​image to obtain the background points and foreground objects of the spliced ​​image; i≥1; S4.1.9 Based on the background points and foreground objects in the i-th frame of the spliced ​​image, a background model of the i+1-th frame of the spliced ​​image is updated using a random update method; S4.1.10 Repeat steps S4.1.8 and S4.1.9 to output the foreground target in each frame of the stitched image, and then obtain the precise angular information of the UAV.

9. The multi-source fusion detection and formation recognition system for UAV clusters according to claim 6 is characterized in that: The method of step S4.2 includes: S4.2.1 Based on the command and control center time, synchronize the preliminary azimuth information and the precise angle information of the UAV to the standard time; using the geocentric coordinate system as the reference coordinate system for spatial alignment, convert the local polar coordinates of the target relative to the radar detection system to the local polar coordinates of the target relative to the photoelectric detection system; S4.2.2 Perform point-track association fusion processing based on the rough angle information in the preliminary orientation information and the precise angle information of the UAV; S4.2.3 The preliminary orientation information and precise angle information of the UAV that are successfully processed by point-track association fusion are compressed and fused through dimensional expansion and lossless transformation of the fusion measurement equation to obtain the three-dimensional position information of the UAV.

10. The multi-source fusion detection and formation recognition system for UAV clusters according to claim 5 is characterized in that: In step S4, the method of identifying the formation of the drone cluster according to the three-dimensional position information of the drone to obtain the formation structure includes: S4.3.1 Use the three-dimensional position information of each drone as input data to model the drone cluster graph; S4.3.2 extract features from the drone cluster graph; the extracted features include connection features, motion features, and node features of the drone cluster graph; S4.3.3 Based on the extracted features, classification and recognition are performed through a graph convolutional neural network, and the formation of the drone cluster after classification and recognition is output.

Images