A UAV swarm target localization method based on the fusion of vision and radar information

Abstract

The present invention discloses a method for locating a target of a swarm of unmanned aerial vehicles (UAVs) by fusing visual and radar information, comprising: utilizing a battlefield surveillance radar to provide real-time battlefield situation information; employing an optimization method to solve the position and pod attitude of our UAVs and dynamically planning the flight path of the swarm of UAVs; rapidly flying to the battlefield according to a preset flight path to perform visual target detection and matching tasks; our UAVs performing real-time detection and processing on the collected images according to a visual target detection algorithm; employing a method of associating and fusing visual and radar information to achieve accurate matching of targets within the field of view; and individually allocating one of our UAVs to each enemy UAV to perform visual tracking and complete the target terminal guidance strike interception task. By associating and fusing multi-source information, the present invention aims to improve the accuracy, real-timeness, and robustness of target association matching and positioning estimation, thereby providing a more reliable solution for the coordinated combat of UAV swarms.

Description

Technical Field

[0001] The present invention relates to the field of machine vision and image processing technology, and in particular to a method for locating unmanned aerial vehicle (UAV) cluster targets by fusing vision and radar information. Background Art

[0002] Unmanned aerial vehicles (UAVs), with their low cost, long endurance, high maneuverability, and strong stealth, are gaining increasing attention in military, civilian, and industrial applications. As the number of UAVs increases, the complexity of UAV swarm coordinated combat systems is increasing. Target association matching and location estimation are key components to ensure the effective operation of UAV swarm systems.

[0003] Traditional UAV swarm collaborative combat methods primarily rely on single sensor information, such as electro-optical / infrared sensors, synthetic aperture radar, and lidar. However, these methods exhibit limitations when faced with complex environments and continuous measurement errors. Furthermore, existing technologies often inefficiently utilize multi-source heterogeneous information, making it difficult to effectively integrate this multi-source information to improve the accuracy and robustness of target association matching and positioning estimation during swarm combat.

[0004] In a UAV swarm collaborative combat system, drones can be equipped with different types of sensors, such as high-resolution cameras, GPS, and airborne radars, while ground-based systems can deploy fixed or mobile battlefield surveillance radars. Therefore, a method that can effectively fuse information from these heterogeneous sensors is urgently needed to improve the reliability of target association matching and positioning estimation, and thus the performance of UAV swarm collaborative combat systems. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a UAV cluster target positioning method that fuses visual and radar information. By associating and fusing multi-source information, it aims to improve the accuracy, real-time and robustness of target association matching and positioning estimation, thereby providing a more reliable solution for the coordinated combat of UAV clusters.

[0006] The present invention solves the technical problem by adopting the following technical solutions:

[0007] A method for swarm target positioning of unmanned aerial vehicles (UAVs) by fusing vision and radar information includes the following steps:

[0008] Step 1: Use battlefield surveillance radar deployed on a mobile radar vehicle or onboard a UAV to detect and track enemy UAV targets and provide real-time battlefield situation information.

[0009] Step 2: When the radar detects an enemy drone cluster, it uses the optimization method to calculate the position and pod attitude of our drones, with the constraint of maximizing the field of view coverage of the cluster targets, and dynamically plans the flight path of the drone cluster.

[0010] Step 3: When the enemy drone cluster reaches the predetermined flight distance, the control station automatically issues a command to dispatch a specified number of our drones to quickly fly to the battlefield along the preset flight path. After arriving at the designated location, the camera immediately opens and performs visual target detection and matching tasks.

[0011] Step 4: Our drone is equipped with a pod and a high-resolution monocular vision camera to collect image information within the field of view. It then performs real-time detection and processing on the collected images based on the visual target detection algorithm, and returns the detection frame information of all targets in the image to the ground control station in real time.

[0012] Step 5: Based on the information returned by the UAV and radar to the ground control station, the radar provides the target's global spatial position coordinates for visual target association matching. The visual and radar information are associated and fused to achieve accurate matching of targets within the field of view, and then the target position is solved for positioning estimation. At the same time, a global number is assigned to the successfully matched target.

[0013] Step 6: Based on the enemy UAV's global number information, our UAV's field of view coverage, and target positioning estimation, assign a our UAV to each enemy UAV to perform visual tracking and complete the target terminal guidance strike interception mission.

[0014] Furthermore, in step S1, before using the battlefield surveillance radar deployed on a mobile radar vehicle or onboard a drone to detect and track enemy drone targets, it also includes: the ground control station interacts and comprehensively dispatches the battlefield surveillance radar and the cluster drones, and realizes the transmission and reception of information between the ground control station and the drone cluster through the two-way communication unit between the ground and the drone.

[0015] Furthermore, in step S1, the real-time battlefield situation information mainly includes the formation of the enemy UAV cluster, the number of UAV targets, the flight speed and real-time position coordinates of the UAVs, and the global position information of our UAVs.

[0016] Furthermore, in step S1, the target detected by the battlefield surveillance radar is represented by an ellipsoid, whose radius is the maximum error in all directions; differential calibration is performed based on the spatial position coordinates of the drone detected by the battlefield surveillance radar and the spatial position coordinates returned by the GPS onboard our drone.

[0017] Furthermore, in step 2, the optimization method can use the particle swarm algorithm to find the optimal solution for maximizing the field of view coverage:

[0018] The input of the particle swarm algorithm mainly includes the enemy drone cluster formation detected by radar, the number of drone targets, the flight speed and real-time position coordinates of the drones, and the total number of our drones;

[0019] The particle swarm algorithm takes maximizing global field of view coverage as the main constraint, and satisfies the number of captured fields of view during the field of view coverage process to avoid the situation where the target is blocked in the same field of view;

[0020] The output of the particle swarm algorithm mainly includes the maximum global field of view coverage, the minimum number of required drones, and the spatial position coordinates and pod attitude angle of each drone.

[0021] Furthermore, in step S4, the visual target detection algorithm adopts a lightweight network structure of the YOLO series, Fast R-CNN or EfficientDet.

[0022] Furthermore, in step S4, the detection frame information of all targets in the image mainly includes the image coordinates, scale, acquisition time, GPS coordinates of the drone, pod posture information and the image of the area where the target is located. There is no need to return to the original overall image to ensure real-time information transmission under limited communication bandwidth conditions.

[0023] Furthermore, in step S5, the method of achieving accurate matching of targets within the field of view by associating and fusing visual and radar information includes:

[0024] Step S51: achieving preliminary global association of enemy UAV targets through bidirectional mapping conversion between radar and visual modal information;

[0025] In step S52 , target association matching is performed on targets that do not have the same number in any two perspective images with overlapping fields of view, and no matching is performed on targets that have the same number.

[0026] Furthermore, in step S51, a preliminary global association of enemy UAV targets is achieved through bidirectional mapping conversion between radar and visual modal information. The steps are as follows:

[0027] Step S511: Based on the spatial position coordinates of our drone provided by the onboard GPS and the attitude information of the camera pod, the spatial position coordinates of the enemy drone detected by the radar are converted into the field of view images of different friendly drones. Then, the coordinates of the enemy drone in the field of view images collected by the camera are compared. If the coordinate distance is less than a threshold, it is considered to be the same enemy drone and the same number is assigned to them, thus achieving the mapping from the spatial coordinates provided by the radar to the visual image coordinates.

[0028] In step S512, based on the coordinate position of the enemy UAV in the image, a ray vector pointing from our UAV to the target UAV is obtained in the image coordinate system; through the spatial position coordinates of our UAV provided by the onboard GPS and the attitude information of the camera pod, the ray vector pointing from our UAV to the target UAV is converted to the world coordinate system, and the set of potential target locations of the target photographed by our UAV within the local range of the radar is estimated, thereby realizing the mapping from the visual image coordinates to the spatial coordinates provided by the radar.

[0029] Furthermore, in step S52, the target association matching method calculates the distance from the camera center to the target and the coordinates of the target in space based on the feature points of the target area in the images captured from the perspectives of the two drone pods and the internal and external parameters of the camera. The steps are as follows:

[0030] Step S521: extract the same key visual feature points in the target area of ​​the two images, use the feature point detection algorithm to obtain multiple sets of coordinate representations for each target, and perform multi-point error calibration in subsequent calculations;

[0031] Step S522 , converting the coordinates of the target multiple feature points in the image coordinate system into coordinates in the camera coordinate system according to the focal length and principal point coordinates of each camera;

[0032] Step S523, converting the coordinates of the target multiple feature points in the camera coordinate system into coordinates in the world coordinate system according to the intrinsic and extrinsic parameter matrices of each camera;

[0033] Step S524: Obtain a ray vector pointing from the user's UAV to the target UAV using the calculated camera optical center and the target's world coordinate point. Then, obtain a ray beam pointing from the user's UAV to the target UAV based on multiple sets of coordinate representations for each target.

[0034] In step S525, the minimum distance between different ray vector groups between the two images is calculated based on the calculated ray beam; if the minimum distance is less than the threshold, the two perspective image targets are determined to be the same target and are assigned the same number; the midpoint of the minimum distance between the two rays is then regarded as the spatial position of the target, and finally the spatial position coordinates of the target are calculated through the calculated multiple groups of world coordinate points to achieve positioning estimation of the cluster target.

[0035] The present invention discloses a method for swarm target positioning of unmanned aerial vehicles (UAVs) by fusing visual and radar information, which has the following beneficial effects:

[0036] (1) Multimodal Information Association: This invention combines radar and a high-resolution monocular vision camera to achieve multimodal information association. By combining radar and visual information, the accuracy of target matching and positioning is improved, making the UAV swarm combat system more adaptable and robust.

[0037] (2) Maximizing the field of view coverage: The ground control station calculates the position and pod attitude of our drones and uses the most optimized method to maximize the field of view coverage of the enemy drone cluster targets. This method enables the drone cluster to more flexibly grasp the formation of the enemy drone cluster targets.

[0038] (3) Automated Scheduling and Deployment: This invention implements the automated scheduling and deployment of UAVs. When an enemy target is within a specified flight distance, the system automatically dispatches a specified number of UAVs to accelerate their arrival at the battlefield and execute the mission, reducing the need for human intervention and improving the system's response speed.

[0039] (4) Assigning global numbers: By assigning global numbers to successfully matched targets, the present invention achieves individual tracking of each enemy UAV during the mission execution phase, thereby improving the system's target strike accuracy and execution efficiency.

[0040] (5) Improvement of comprehensive combat capability: Through the above innovations, the present invention enables drone clusters to cooperate in combat, achieve accurate monitoring, tracking and attack on enemy drone clusters, improve comprehensive combat capability, and make the system more suitable for complex and dynamic battlefield environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flow chart of the method of the present invention.

[0042] Figure 2 Schematic diagram for maximizing field of view coverage.

[0043] Figure 3 Schematic diagram of the correlation and fusion of visual and radar information. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0045] In order to address the high strike repetition rate and miss rate issues in existing rule-based cluster target strike methods, the present invention uses a method based on the fusion of visual and radar information to perform global target association matching, thereby realizing multi-target positioning of enemy clusters during joint operations by drone clusters. Specifically, the drone carries a high-resolution monocular visual camera to collect field image information under the conditions of the current drone position and pod attitude, and the ground battlefield surveillance radar provides the enemy drone cluster's formation and distance information. With the assistance of radar information, visual target association matching is performed, and the same global number is assigned to successfully matched targets, realizing multi-machine and multi-target collaborative positioning estimation, and ultimately improving the target terminal guidance strike success rate.

[0046] refer to Figure 1 The present invention discloses a method for locating a target of a UAV cluster by integrating visual and radar information, comprising the following steps:

[0047] Step 1: The ground control station interacts and coordinates the battlefield surveillance radar and swarm drones. Information is sent and received between the ground control station and the drone swarm via two-way communication units between the ground and drones. Battlefield surveillance radars deployed on mobile radar vehicles or onboard drones are used to detect and track enemy drone targets, providing real-time battlefield situation information, including the enemy drone swarm's formation, number of drone targets, flight speed, and real-time location coordinates, as well as the global location of our own drones. Upon detecting a suspicious enemy target swarm, the ground control station dispatches a battlefield surveillance radar vehicle toward the target swarm to more accurately obtain information about the enemy drone target swarm.

[0048] Step 2: When the radar detects an enemy drone cluster, it uses the optimization method to calculate the position and pod attitude of our drones, with the constraint of maximizing the field of view coverage of the cluster targets, and dynamically plans the flight path of the drone cluster.

[0049] Step 3: When the enemy drone cluster reaches the predetermined flight distance, the control station automatically issues a command to dispatch a specified number of our drones to quickly fly to the battlefield along the preset flight path. After arriving at the designated location, the camera immediately opens and performs visual target detection and matching tasks.

[0050] Step 4: Our drone is equipped with a pod and a high-resolution monocular vision camera to collect image information within the field of view. It then performs real-time detection and processing on the collected images based on the visual target detection algorithm, and returns the detection frame information of all targets in the image to the ground control station in real time.

[0051] Step 5: Based on the information returned by the UAV and radar to the ground control station, the radar provides the target's global spatial position coordinates for visual target association matching. The visual and radar information are associated and fused to achieve accurate matching of targets within the field of view, and then the target position is solved for positioning estimation. At the same time, a global number is assigned to the successfully matched target.

[0052] Step 6: Based on the enemy UAV's global number information, our UAV's field of view coverage, and target positioning estimation, assign a our UAV to each enemy UAV to perform visual tracking and complete the target terminal guidance strike interception mission.

[0053] Furthermore, in step S1, the battlefield surveillance radar is used to detect the global position information of friendly UAVs and enemy UAVs in the battlefield:

[0054] Due to errors in radar measurement, the detected target is represented by an ellipsoid, whose radius is the maximum error in all directions; differential calibration is performed based on the spatial position coordinates of the drone detected by the battlefield surveillance radar and the spatial position coordinates returned by the GPS onboard our drone.

[0055] Furthermore, in step 2, the optimization method can use the particle swarm algorithm to find the optimal solution for maximizing the field of view coverage:

[0056] The input of the particle swarm algorithm mainly includes the enemy drone cluster formation detected by radar, the number of drone targets, the flight speed and real-time position coordinates of the drones, and the total number of our drones;

[0057] The particle swarm algorithm takes maximizing global field of view coverage as its main constraint. During the field of view coverage process, the number of captured fields of view, that is, the minimum number of our drones required, should be less than or equal to the total number of our drones, and the target should be avoided from being blocked in the same field of view as much as possible.

[0058] The output of the particle swarm algorithm mainly includes the maximum global field of view coverage, the minimum number of required drones, and the spatial position coordinates and pod attitude angle of each drone.

[0059] Furthermore, in step S4, the image captured by the monocular vision camera is detected and processed using a visual target detection algorithm:

[0060] (1) In response to different lighting conditions during the day and at night, the drone carries visible light and infrared monocular vision cameras respectively to meet the requirements of all-day combat missions.

[0061] (2) The visual target detection algorithm uses lightweight network structures such as the YOLO series, Fast R-CNN, and EfficientDet, which have small parameters and fast speed, and can be easily deployed on the mobile airborne terminal of drones.

[0062] (3) Due to the large distance from the enemy drone target, the target scale in the captured image is small. The embedded airborne visual detection algorithm needs to be optimized for the weak target features to ensure the accuracy of target detection. Furthermore, in step S4, the detection frame information of all targets in the image mainly includes the image coordinates of the target, scale, acquisition time, GPS coordinates of the drone, pod posture information, and the image of the target area. There is no need to return to the original entire image to ensure real-time information transmission under limited communication bandwidth conditions.

[0063] (4) The detection information returned by the airborne end to the ground control station mainly includes the target's image coordinates, scale, acquisition time, GPS coordinates of the drone, pod attitude information and the image of the target area. There is no need to return the original overall image to ensure real-time information transmission under limited communication bandwidth conditions.

[0064] Furthermore, in step S5, the method of achieving accurate matching of targets within the field of view by associating and fusing visual and radar information includes:

[0065] (1) Through the bidirectional mapping conversion between radar and visual modal information, the global association of enemy UAV preliminary targets is achieved.

[0066] (a) Based on the spatial position coordinates of the friendly UAV provided by the onboard GPS and the attitude information of the camera pod, the spatial position coordinates of the enemy UAV detected by the radar are converted to the field of view images of different friendly UAVs. The coordinates are then compared with the image coordinates of the enemy UAV in the field of view images collected by the camera. If the coordinate distance is less than a threshold, they are considered to be the same enemy UAV and are assigned the same number, thus achieving the mapping from the spatial coordinates provided by the radar to the visual image coordinates.

[0067] (b) Based on the enemy drone's coordinate position in the image, a ray vector pointing from the friendly drone to the target drone in the image coordinate system can be derived. Using the friendly drone's spatial position coordinates provided by the onboard GPS and the camera pod's attitude information, the ray vector pointing from the friendly drone to the target drone is converted to the world coordinate system. This estimates the set of potential target locations within the radar's local range captured by the friendly drone, achieving a mapping from visual image coordinates to the spatial coordinates provided by the radar.

[0068] (c) In steps (a) and (b) above, the bidirectional mapping conversion process between radar and vision can be calculated in parallel to reduce time overhead. By taking the intersection of the two association matching results through bidirectional mapping conversion, the probability of target association matching errors in unidirectional mapping can be effectively reduced.

[0069] (2) In order to further improve the success rate of target association matching and effectively reduce the target missed matching rate in the bidirectional mapping conversion process between radar and vision, target association matching is performed on targets that do not have the same number in any two perspective images with overlapping fields of view, and no matching is performed on targets that have already obtained the same number.

[0070] This target association matching method is based on the feature points of the target area in the images captured from the perspectives of two drone pods. The distance from the camera center to the target and the target's coordinates in space are calculated using the internal and external parameters of the camera. The steps are as follows:

[0071] (a) Extract the same key visual feature points in the target area of ​​the two images. Feature point detection algorithms such as ORB and SIFT can be used to obtain multiple sets of coordinate representations for each target, and perform multi-point error calibration in subsequent calculations.

[0072] (b) According to the focal length and principal point coordinates of each camera, the coordinates of multiple sets of feature points of the target in the image coordinate system are converted into coordinates in the camera coordinate system.

[0073] (c) According to the intrinsic and extrinsic parameter matrices of each camera, the coordinates of multiple groups of target feature points in the camera coordinate system are converted into coordinates in the world coordinate system.

[0074] (d) The ray vector pointing from our UAV to the target UAV is obtained by calculating the camera optical center and the world coordinate point of the target. Then, based on the multiple sets of coordinate representations of each target, the ray beam pointing from our UAV to the target UAV is obtained.

[0075] (e) Based on the calculated ray bundles, the minimum distance between different ray vector groups between the two images is further calculated. If the minimum distance is less than a threshold, the targets in the two viewpoint images are determined to be the same target and are assigned the same number. The midpoint of the minimum distance between the two rays is then considered the spatial position of the target. Finally, the spatial position coordinates of the target are calculated using the multiple sets of world coordinate points, achieving the location estimation of the clustered targets.

[0076] The present invention discloses a method for locating target of UAV clusters by integrating vision and radar information, which can achieve:

[0077] (1) Multimodal Information Association: This invention combines radar and a high-resolution monocular vision camera to achieve multimodal information association. By combining radar and visual information, the accuracy of target matching and positioning is improved, making the UAV swarm combat system more adaptable and robust.

[0078] (2) Maximizing the field of view coverage: The ground control station calculates the position and pod attitude of our drones and uses the most optimized method to maximize the field of view coverage of the enemy drone cluster targets. This method enables the drone cluster to more flexibly grasp the formation of the enemy drone cluster targets.

[0079] (3) Automated Scheduling and Deployment: This invention implements the automated scheduling and deployment of UAVs. When an enemy target is within a specified flight distance, the system automatically dispatches a specified number of UAVs to accelerate their arrival at the battlefield and execute the mission, reducing the need for human intervention and improving the system's response speed.

[0080] (4) Assigning global numbers: By assigning global numbers to successfully matched targets, the present invention achieves individual tracking of each enemy UAV during the mission execution phase, thereby improving the system's target strike accuracy and execution efficiency.

[0081] (5) Improvement of comprehensive combat capability: Through the above innovations, the present invention enables drone clusters to cooperate in combat, achieve accurate monitoring, tracking and attack on enemy drone clusters, improve comprehensive combat capability, and make the system more suitable for complex and dynamic battlefield environments.

[0082] Example

[0083] Target association, matching, and location estimation primarily rely on a ground control station (GCS), battlefield surveillance radar, UAVs equipped with high-resolution visual cameras, and a two-way communication unit. The GCS is primarily responsible for the overall command and control of UAV swarm joint operations, including autonomous control of the radar and UAVs, as well as the interpretation of the return information transmitted by the radar and UAVs. The battlefield surveillance radar is primarily used for efficient global detection of battlefield targets and the environment, providing target position, velocity, and other motion information. The UAV swarm is primarily used for identifying, tracking, real-time monitoring, and engaging enemy UAV targets. By capturing high-resolution images of a local field of view and transmitting target information to the GCS in real time, this provides more detailed information for subsequent target association. The two-way communication unit primarily facilitates two-way information transmission between the ground and UAVs. In short, the GCS, battlefield surveillance radar, UAVs equipped with high-resolution visual cameras, and the two-way communication unit form a collaborative system that achieves global perception, identification, tracking, and engagement of targets on the battlefield. This multimodal data fusion and collaboration helps improve the accuracy and real-time performance of target association.

[0084] Assume that the number of drones contained in the enemy drone swarm is M, and the number of available drones on our side is N, where the number of our drones is not less than the number of enemy drones, and usually satisfies N ≥ 1.5M. The following is a detailed description of a drone swarm target positioning method based on the fusion of visual and radar information, including the following steps:

[0085] Step 1: The radar monitors the battlefield in real time. When a suspicious enemy target group is detected, the ground control station sends a battlefield surveillance radar vehicle to move towards the target cluster to obtain more accurate enemy drone target cluster information.

[0086] (1) The radar detection target is represented by an ellipsoid, and the length of its semi-axis is the maximum error in each direction. Any point in the ellipsoid represents the possible location of the target. Its probability distribution is a three-dimensional standard normal distribution with the center of the sphere as the coordinate center origin, a mean of 0, and a standard deviation of the maximum error value. Its probability density function is Where x is the target's coordinate position within the ellipsoid, and Σ represents the covariance matrix. Furthermore, considering the small size of the target, which leads to errors in radar detection, the number of enemy drone targets detected by the radar should be M ± ε, where M represents the actual number of enemy drones and ε represents the error in the number of targets detected by the radar.

[0087] (2) In order to further improve the radar detection accuracy, the spatial position coordinates (X r ,Y r ,Z r ) and the spatial position coordinates returned by our drone’s onboard GPS (X GPS ,Y GPS ,Z GPS ) for differential calibration.

[0088]

[0089] Among them, (ΔX, ΔY, ΔZ) is the error that needs to be calibrated, n is the number of data points of our drone, and i represents the i-th data point.

[0090] Step 2: The ground control station uses the optimization method to calculate the position and pod attitude of our drones, with the constraint of maximizing the field of view coverage of the enemy drone cluster targets, and dynamically plans the flight path of the drone cluster.

[0091] (1) The input of the particle swarm algorithm mainly includes the enemy UAV cluster formation (including straight line, V-shape, diamond, circle, etc.) detected by radar, number, flight speed, real-time position coordinates, and the total number of our UAVs.

[0092] (2) The particle swarm algorithm takes maximizing global field of view coverage as the main constraint (e.g. Figure 2 As shown in the figure), during the field of view coverage process, the required number of fields of view V (i.e. the required number of drones) should be less than or equal to the total number of our drones V≤N, and the target should be avoided from being blocked in the same field of view as much as possible.

[0093] (3) The output of the particle swarm algorithm mainly includes the maximum global field of view coverage (Max_Coverage), the minimum number of drones required V min , and the spatial position coordinates and pod attitude angles of each drone {(P1,Θ1),(P2,Θ2),...,(P V ,Θ V )}. Among them, P i represents the coordinate position of the i-th UAV of our side, Θ i Represents the attitude angle of the pod of the i-th friendly UAV.

[0094] Step 3: When the enemy drone cluster reaches the specified flight distance D, the control station automatically issues a command to dispatch a specified number of friendly drones, which fly quickly to the battlefield according to the preset flight path. After arriving at the designated location, the camera immediately opens and performs tasks such as visual target detection, association matching, and positioning estimation.

[0095] Step 4: Our drone carries a pod and a high-resolution monocular vision camera, performs real-time detection and processing on the collected images based on the visual target detection algorithm, and returns the detection frame information of all targets in the image to the ground control station in real time.

[0096] (1) In response to different lighting conditions during the day and at night, the drone carries visible light and infrared monocular vision cameras respectively to meet the requirements of all-day combat missions.

[0097] (2) The visual target detection algorithm uses lightweight network structures such as the YOLO series, Fast R-CNN, and EfficientDet, which have small parameters and fast speed, and can be easily deployed on the mobile airborne terminal of drones.

[0098] (3) Since the enemy UAV target is far away and the target scale in the collected image is small, the embedded airborne visual detection algorithm needs to be optimized for the weak target features to ensure the target detection accuracy.

[0099] (4) The detection information returned by the airborne end to the ground control station mainly includes the target's image coordinates, scale, acquisition time, GPS coordinates of the drone, pod attitude information and the image of the target area. There is no need to return the original overall image to ensure real-time information transmission under limited communication bandwidth conditions.

[0100] Step 5: Based on the information returned by the UAV and radar to the ground control station, the radar provides the global spatial position coordinates of the target for visual matching, and the visual and radar information are integrated (such as Figure 3 ), achieving precise matching of targets within the field of view and assigning global numbers to successfully matched targets.

[0101] (1) Through the bidirectional mapping conversion between radar and visual modal information, the global association of enemy UAV preliminary targets is achieved.

[0102] (a) Based on the spatial position coordinates of the friendly UAV provided by the onboard GPS and the attitude information of the camera pod, the spatial position coordinates of the enemy UAV detected by the radar are converted to the field of view images of different friendly UAVs. The coordinates are then compared with the image coordinates of the enemy UAV in the field of view images collected by the camera. If the coordinate distance is less than a threshold, they are considered to be the same enemy UAV and are assigned the same number, thus achieving the mapping from the spatial coordinates provided by the radar to the visual image coordinates.

[0103] (b) Based on the enemy drone's coordinate position in the image, a ray vector pointing from the friendly drone to the target drone in the image coordinate system can be derived. Using the friendly drone's spatial position coordinates provided by the onboard GPS and the camera pod's attitude information, the ray vector pointing from the friendly drone to the target drone is converted to the world coordinate system. This estimates the set of potential target locations within the radar's local range captured by the friendly drone, achieving a mapping from visual image coordinates to the spatial coordinates provided by the radar.

[0104] (c) In steps (a) and (b) above, the bidirectional mapping conversion process between radar and vision can be performed in parallel to reduce time overhead. By taking the intersection of the two matching results through bidirectional mapping, the probability of incorrect target matching in unidirectional mapping can be effectively reduced.

[0105] (2) In order to further improve the success rate of target association matching and effectively reduce the target missed matching rate in the bidirectional mapping conversion process between radar and vision, target association matching is performed on targets that do not have the same number in any two images with overlapping fields of view, and no matching is performed on targets that have already obtained the same number. This target association matching method is based on the feature points of the target area in the images collected from the perspectives of the two drone pods. The distance from the camera center to the target and the coordinates of the target in space are calculated through the internal and external parameters of the camera. The steps are as follows:

[0106] (a) Use feature point detection algorithms such as ORB and SIFT to extract the key visual feature points of the target area in the two images, calculate the descriptor of each feature point, and use nearest neighbor matching for feature matching. In this way, multiple sets of coordinate representations of each target are obtained. To perform multi-point error calibration in subsequent calculations, where (u m ,v m ) represents the mth group of coordinates of the target, u m and v m Respectively represent the horizontal and vertical coordinates in the image coordinate system.

[0107] (b) According to the focal length f of each camera x ,f y and the principal point coordinates (c x ,c y ), the coordinates of multiple sets of target feature points in the image coordinate system (u j ,v j ) (Take the jth group of coordinates as an example) to convert to the coordinates in the camera coordinate system (x c ,y c ,z c ). Then we have:

[0108]

[0109] (c) According to the external parameter matrix [R|T] of each camera (i.e., the combination of rotation matrix and translation vector), the coordinates (x c ,y c ,z c ) is converted to the coordinates in the world coordinate system (x w ,y w ,z w ), where the coordinates of each camera center O in the world coordinate system are: Then we have:

[0110]

[0111] (d) Obtain the ray vector from our drone to the target drone by calculating the camera optical center and the target's world coordinate point Where λ∈[0,∞) represents the slope of the ray. Then, based on the multiple sets of coordinates of each target, the ray beam pointing from our UAV to the target UAV is obtained. Among them, P m represents the mth ray in the ray bundle.

[0112] (e) Based on the calculated ray bundles, further calculate the minimum distance between different ray vector groups between the two images. If the minimum distance is less than the threshold, the two viewpoint image targets are determined to be the same target and are assigned the same number. The midpoint of the minimum distance between the two rays is then regarded as the spatial position of the target (x′ w ,y′ w ,z′ w ), and finally calculate the geometric center point of the target in space through the calculated multiple sets of world coordinate points, which is the spatial position of the target Achieve localization estimation of cluster targets.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for swarm target positioning of UAVs by integrating vision and radar information, characterized in that: The steps include: Step 1: Use battlefield surveillance radar deployed on mobile radar vehicles or UAVs to detect and track enemy UAV targets and provide real-time battlefield situation information; Step 2: When the radar detects an enemy drone swarm, it uses an optimization method to calculate the position and pod attitude of our drones, with the constraint of maximizing the field of view coverage of the swarm targets, and dynamically plans the flight path of the drone swarm; Step 3: When the enemy drone swarm reaches the predetermined flight distance, the control station automatically issues a command to dispatch a specified number of friendly drones, which fly rapidly to the battlefield along the preset flight path. Upon reaching the designated location, the cameras immediately open and perform visual target detection and matching tasks. Step 4: Our drone is equipped with a pod and a high-resolution monocular vision camera to collect image information within the field of view. It then performs real-time detection and processing on the collected images based on the visual target detection algorithm, and returns the detection frame information of all targets in the image to the ground control station in real time. Step 5: Based on the information returned by the UAV and radar to the ground control station, the radar provides the target's global spatial position coordinates for visual target association matching. By integrating visual and radar information, accurate target matching within the field of view is achieved, and the target position is calculated for positioning estimation. At the same time, a global number is assigned to the successfully matched target. Step 6: Based on the enemy drone's global number information, our drone's field of view coverage, and target location estimation, assign a our drone to each enemy drone to perform visual tracking and complete the target terminal guidance strike interception mission; In step S5, the method of achieving accurate matching of targets within the field of view by associating and fusing visual and radar information includes: Step S51: achieving preliminary global association of enemy UAV targets through bidirectional mapping conversion between radar and visual modal information; In step S52 , target association matching is performed on targets that do not have the same number in any two perspective images with overlapping fields of view, and no matching is performed on targets that have the same number.

2. The method for locating target of UAV clusters by integrating vision and radar information according to claim 1 is characterized in that: In step S1, before using the battlefield surveillance radar deployed on a mobile radar vehicle or onboard a drone to detect and track enemy drone targets, it also includes: the ground control station interacts and comprehensively dispatches the battlefield surveillance radar and the cluster drones, and realizes the transmission and reception of information between the ground control station and the drone cluster through the two-way communication unit between the ground and the drone.

3. The method for locating target of UAV clusters by integrating vision and radar information according to claim 2 is characterized in that: In step S1, the real-time battlefield situation information mainly includes the enemy UAV cluster formation, the number of UAV targets, the flight speed and real-time position coordinates of the UAVs, and the global position information of our UAVs.

4. The method for locating target of UAV clusters by integrating vision and radar information according to claim 3 is characterized in that: In step S1, the target detected by the battlefield surveillance radar is represented by an ellipsoid, whose radius is the maximum error in all directions; differential calibration is performed based on the spatial position coordinates of the drone detected by the battlefield surveillance radar and the spatial position coordinates returned by the GPS onboard our drone.

5. The method for locating target of UAV clusters by integrating vision and radar information according to claim 4 is characterized in that: In step 2, the optimization method uses a particle swarm algorithm to find the optimal solution that maximizes the field of view coverage: The input of the particle swarm algorithm mainly includes the enemy drone cluster formation detected by radar, the number of drone targets, the flight speed and real-time position coordinates of the drones, and the total number of our drones; The particle swarm algorithm takes maximizing global field of view coverage as the main constraint, and satisfies the number of captured fields of view during the field of view coverage process to avoid the situation where the target is blocked in the same field of view; The output of the particle swarm algorithm mainly includes the maximum global field of view coverage, the minimum number of required drones, and the spatial position coordinates and pod attitude angle of each drone.

6. The method for locating target of UAV clusters by integrating vision and radar information according to claim 5 is characterized in that: In step S4, the visual target detection algorithm adopts a lightweight network structure of the YOLO series, Fast R-CNN or EfficientDet.

7. The method for locating target of UAV clusters by integrating vision and radar information according to claim 6 is characterized in that: In step S4, the detection frame information of all targets in the image mainly includes the image coordinates, scale, acquisition time, GPS coordinates of the drone, pod posture information and the image of the area where the target is located. There is no need to return to the original overall image to ensure real-time information transmission under limited communication bandwidth conditions.

8. The method for locating target of UAV clusters by integrating vision and radar information according to claim 7 is characterized in that: In step S51, preliminary global association of enemy UAV targets is achieved through bidirectional mapping conversion between radar and visual modal information. The steps are as follows: Step S511: Based on the spatial position coordinates of our drone provided by the onboard GPS and the attitude information of the camera pod, the spatial position coordinates of the enemy drone detected by the radar are converted into the field of view images of different friendly drones. Then, the coordinates of the enemy drone in the field of view images collected by the camera are compared. If the coordinate distance is less than a threshold, it is considered to be the same enemy drone and the same number is assigned to them, thus achieving the mapping from the spatial coordinates provided by the radar to the visual image coordinates. In step S512, based on the coordinate position of the enemy UAV in the image, a ray vector pointing from our UAV to the target UAV is obtained in the image coordinate system; through the spatial position coordinates of our UAV provided by the onboard GPS and the attitude information of the camera pod, the ray vector pointing from our UAV to the target UAV is converted to the world coordinate system, and the set of potential target locations of the target photographed by our UAV within the local range of the radar is estimated, thereby realizing the mapping from the visual image coordinates to the spatial coordinates provided by the radar.

9. The method for locating target of UAV clusters by integrating vision and radar information according to claim 8 is characterized in that: In step S52, the target association matching method calculates the distance from the camera center to the target and the coordinates of the target in space based on the feature points of the target area in the images captured from the perspectives of the two drone pods and the internal and external parameters of the camera. The steps are as follows: Step S521: extract the same key visual feature points in the target area of ​​the two images, use the feature point detection algorithm to obtain multiple sets of coordinate representations for each target, and perform multi-point error calibration in subsequent calculations; Step S522 , converting the coordinates of the target multiple feature points in the image coordinate system into coordinates in the camera coordinate system according to the focal length and principal point coordinates of each camera; Step S523, converting the coordinates of the target multiple feature points in the camera coordinate system into coordinates in the world coordinate system according to the intrinsic and extrinsic parameter matrices of each camera; Step S524: Obtain a ray vector pointing from the user's UAV to the target UAV using the calculated camera optical center and the target's world coordinate point. Then, obtain a ray beam pointing from the user's UAV to the target UAV based on multiple sets of coordinate representations for each target. In step S525, the minimum distance between different ray vector groups between the two images is calculated based on the calculated ray beam; if the minimum distance is less than the threshold, the two perspective image targets are determined to be the same target and are assigned the same number; the midpoint of the minimum distance between the two rays is then regarded as the spatial position of the target, and finally the spatial position coordinates of the target are calculated through the calculated multiple groups of world coordinate points to achieve positioning estimation of the cluster target.

Images