Target matching method between multi-station visual intersection array and UAV cascade radar

Abstract

Target matching method of multi-station visual intersection array and UAV cascade radar. Belongs to the field of target detection and matching technology. When using a multi-station visual intersection array to track UAVs, the camera at each angle will capture several different UAV targets. At this time, accurate matching of the targets captured by the cameras at each angle becomes a technical problem. The present invention can solve this problem. By determining the number of the UAV targets in the cascade radar, the UAV targets in the multi-angle photos taken in the multi-station visual intersection array are accurately numbered. In this way, in the subsequent processing process, it can be determined which numbered UAV each target in each angle photo in the multi-station visual intersection array specifically belongs to, so that when the visual intersection array composed of multiple cameras is used to perform triangulation positioning and image restoration on any target in the UAV cluster, it can be accurately matched to avoid target tracking errors.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and matching, and specifically relates to the technical field of target matching between a multi-station visual intersection array and an unmanned aerial vehicle cascade radar. Background Art

[0002] With the rapid development of unmanned aerial vehicle (UAV) technology, UAV fleets have shown tremendous potential for application in military reconnaissance, disaster monitoring, logistics and transportation, agricultural plant protection, geographic surveying and mapping, and other fields. Efficient and accurate tracking of UAV fleets is a key technical foundation for ensuring the safety of UAV collaborative operations and enabling complex mission planning and control.

[0003] Optical cameras (visible light / infrared) are used to image and track drone targets. Vision systems have high angular resolution and can provide rich information about target appearance features (such as shape and texture), facilitating target recognition. However, monocular vision struggles to accurately capture the target's three-dimensional position, necessitating a multi-camera visual intersection array (VIA) for triangulation. However, when using a multi-station VIA for drone tracking, each camera at each angle will capture several different drone targets, making accurate matching of the targets captured by each camera angle a technical challenge. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a target matching method for a multi-station visual intersection array and a UAV cascade radar, which helps to accurately match the targets photographed by cameras at various angles in the multi-station visual intersection array through radar monitoring.

[0005] The method comprises the following steps:

[0006] S1, the multi-station visual intersection array uses N cameras to obtain N two-dimensional images of the UAV cluster at time t from different directions;

[0007] S2. In the drone cluster, M drones are set up and numbered from drone 1 to drone M in sequence. Each drone carries an airborne radar to form a cascade radar. Each radar obtains a 3D point cloud image of the drone cluster at time t, and a total of M 3D point cloud images are obtained;

[0008] S3, using the target recognition model to detect several drone targets and the two-dimensional detection frame corresponding to each drone target from each two-dimensional image;

[0009] S4, using the point cloud target detection model to detect M-1 drone targets and the corresponding 3D detection box of each drone target from each 3D point cloud image;

[0010] S5. For each two-dimensional image, compare the target association degree with any 1-2 3D point cloud images in sequence to obtain the drone number corresponding to each target in each two-dimensional image.

[0011] Furthermore, N is greater than or equal to 3.

[0012] Furthermore, the 3D point cloud image of the drone cluster at time t obtained by the drone carrying the airborne radar contains a total of M-1 drone targets, and each drone target carries a corresponding number. Among the M-1 drone targets, the drone carrying the airborne radar is not included.

[0013] Furthermore, the target recognition model uses the YOLO series model.

[0014] Furthermore, the point cloud target detection model uses the PointPillar point cloud target detection model.

[0015] Furthermore, in step S5, if any two-dimensional image has H drone targets, the following operations are performed:

[0016] S61. Take a 3D point cloud image containing M-1 numbered drone targets;

[0017] S62. For the first numbered UAV target, calculate the degree of correlation between it and the H UAV targets in the two-dimensional image respectively, and select the target with the highest correlation degree among the H UAV targets. The number of the UAV corresponding to this target is the same as the number corresponding to the first UAV target. If the degree of correlation between the first numbered UAV target and the H UAV targets in the two-dimensional image is less than 0.5, it is considered that there is no UAV target in the H UAV targets in the two-dimensional image with the same number as the first numbered UAV target.

[0018] S63: Replace the next numbered drone target and repeat the operation in step S62 until all H drone targets in the two-dimensional image have been numbered. If there are still unnumbered drone targets in the two-dimensional image, proceed to step S64.

[0019] S64. Take another 3D point cloud image, and take a drone target with a different number from that in the first 3D point cloud image. The number corresponding to this target is the number of the drone target that has not been numbered in the two-dimensional image.

[0020] Furthermore, the degree of association between the drone target in the 3D point cloud image and the drone target in the two-dimensional image is calculated as follows: the camera and airborne radar are calibrated with external parameters, the three-dimensional detection frame in the 3D point cloud image is converted from the point cloud coordinate system to the image coordinate system, and the corresponding two-dimensional point cloud detection frame is obtained. The intersection-and-union ratio between the two-dimensional point cloud detection frame and the two-dimensional detection frame of the drone target in the two-dimensional image is calculated, which is the degree of association.

[0021] The beneficial effect of the method described in the present invention is: by determining the number of the drone target in the cascade radar, the drone targets in the multi-angle photos taken in the multi-station visual intersection array are accurately numbered. In this way, in the subsequent processing process, it is possible to determine which numbered drone each target in each angle photo in the multi-station visual intersection array belongs to, so that when the visual intersection array composed of multiple cameras is used to perform triangulation positioning and image restoration on any target in the drone cluster, accurate matching can be achieved, avoiding target tracking errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flowchart of the method in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] Example 1

[0025] This embodiment provides a target matching method between a multi-station visual intersection array and a UAV cascade radar. By determining the number of the UAV target in the cascade radar, the UAV targets in the multi-angle photos taken in the multi-station visual intersection array are accurately numbered. In this way, in the subsequent processing process, it is possible to determine which numbered UAV each target in each angle photo in the multi-station visual intersection array belongs to, so that when the visual intersection array composed of multiple cameras is used to perform triangulation positioning and image restoration on any target in the UAV cluster, accurate matching can be achieved, avoiding target tracking errors.

[0026] like Figure 1 As shown, the method of the present invention comprises the following steps:

[0027] S1, the multi-station visual intersection array uses N cameras to obtain N two-dimensional images of the UAV cluster at time t from different directions;

[0028] S2. In the drone cluster, M drones are set up and numbered from drone 1 to drone M in sequence. Each drone carries an airborne radar to form a cascade radar. Each radar obtains a 3D point cloud image of the drone cluster at time t, and a total of M 3D point cloud images are obtained;

[0029] S3, using the target recognition model to detect several drone targets and the two-dimensional detection frame corresponding to each drone target from each two-dimensional image;

[0030] S4, using the point cloud target detection model to detect M-1 drone targets and the corresponding 3D detection box of each drone target from each 3D point cloud image;

[0031] S5. For each two-dimensional image, compare the target association degree with any 1-2 3D point cloud images in sequence to obtain the drone number corresponding to each target in each two-dimensional image.

[0032] A multi-station visual intersection array usually uses cameras with at least three viewing angles to shoot at different angles, so N is greater than or equal to 3.

[0033] The 3D point cloud image of the drone cluster at time t obtained by the drone carrying the airborne radar contains a total of M-1 drone targets, and each drone target carries a corresponding number. The drone carrying the airborne radar is not included in the M-1 drone targets.

[0034] The target recognition model uses the YOLO series model, and it has been verified that YOLOv11 is the best choice.

[0035] The point cloud target detection model uses the PointPillar point cloud target detection model.

[0036] In step S5, assume that there are H drone targets in any two-dimensional image and perform the following operations:

[0037] S51, take a 3D point cloud image containing M-1 numbered drone targets;

[0038] S52. For the first numbered UAV target, calculate the degree of correlation between it and the H UAV targets in the two-dimensional image respectively, and select the target with the highest correlation degree among the H UAV targets. The number of the UAV corresponding to this target is the same as the number corresponding to the first UAV target. If the degree of correlation between the first numbered UAV target and the H UAV targets in the two-dimensional image is less than 0.5, it is considered that there is no UAV target in the H UAV targets in the two-dimensional image with the same number as the first numbered UAV target.

[0039] S53: Replace the next numbered drone target and repeat the operation in step S52 until all H drone targets in the two-dimensional image have been numbered. If there are still unnumbered drone targets in the two-dimensional image, proceed to step S54.

[0040] S54. Take another 3D point cloud image, and take the drone target with a different number from that in the first 3D point cloud image. The number corresponding to this target is the number of the drone target that has not been numbered in the two-dimensional image.

[0041] The degree of association between the drone target in the 3D point cloud image and the drone target in the 2D image is calculated as follows: extrinsic calibration is performed on the camera and airborne radar, the 3D detection frame in the 3D point cloud image is converted from the point cloud coordinate system to the image coordinate system, and the corresponding 2D point cloud detection frame is obtained. The intersection-and-union ratio between the 2D point cloud detection frame and the 2D detection frame of the drone target in the 2D image is calculated, which is the degree of association.

[0042] Example 2

[0043] This embodiment is a further limitation of embodiment 1. The joint calibration of the camera and the lidar is to obtain the transformation relationship between the camera coordinate system and the point cloud coordinate system, that is, the rotation and translation relationship between the two three-dimensional coordinate systems. The extrinsic parameters between the two sensors are recorded as [ , ],in is the rotation matrix, is the translation matrix, and the formula is as follows:

[0044] ;

[0045] in, is the point coordinate in the point cloud coordinate system, is the point coordinate in the camera coordinate system. The conversion relationship from the point cloud coordinate system to the pixel coordinate system is as follows:

[0046] ;

[0047] It can be further concluded that: ;

[0048] in, is the corresponding pixel point in the pixel coordinate system corresponding to the camera, The origin of the pixel coordinate system corresponding to the camera, , , is the camera focal length, and is the number of pixels in the row and column directions in the image coordinate system corresponding to the camera; 、 and are all 1×4 transformation matrices, let For the augmented matrix, there are 12 unknowns, so at least six pairs of matching points are required during the calibration process to achieve the augmented matrix Linear solution of .

[0049] Using the above method, the 3D detection frame in the 3D point cloud image is converted from the point cloud coordinate system to the image coordinate system. The intersection-and-union ratio between the 2D point cloud detection frame and the 2D detection frame of the drone target in the 2D image is calculated. The intersection-and-union ratio indicates the degree of match between the two rectangular detection frames. The better the match, the higher the degree of association. The degree of association is used to determine whether the two target frames point to the same object.

[0050] Example 3

[0051] This embodiment further limits the embodiment 1 and further limits step S5.

[0052] Case 1: Assume that in a drone flight mission, there are a total of 4 drones, numbered UAV 1, UAV 2, UAV 3 and UAV 4. Correspondingly, the 3D point cloud image taken by the onboard radar of UAV No. 1 contains 3 drone targets numbered 2, 3 and 4, the 3D point cloud image taken by the onboard radar of UAV No. 2 contains 3 drone targets numbered 1, 3 and 4, the 3D point cloud image taken by the onboard radar of UAV No. 3 contains 3 drone targets numbered 1, 2 and 4, and the 3D point cloud image taken by the onboard radar of UAV No. 4 contains 3 drone targets numbered 1, 2 and 3.

[0053] Consider a 2D image captured by a camera in a multi-station visual intersection array. The image contains two drone targets to be numbered (assuming they are drone targets 3 and 4). Now, number the targets in this 2D image:

[0054] Take a 3D point cloud image (assuming it is a 3D point cloud image taken by the onboard radar of UAV No. 1, in which UAVs No. 2, No. 3 and No. 4 are present). For UAV No. 2, the degree of correlation between it and the two UAV targets in the two-dimensional image is calculated respectively. If the correlation degrees are both less than 0.5, it is considered that there is no UAV target in the two-dimensional image with the same number as UAV No. 2. Replace the next numbered UAV target, namely UAV target No. 3, and calculate the degree of correlation between it and the two UAV targets in the two-dimensional image respectively. After calculation, one target in the two-dimensional image is numbered No. 3. Replace the next numbered UAV target, namely UAV target No. 4, and calculate the degree of correlation between it and the two UAV targets in the two-dimensional image respectively. After calculation, one target in the two-dimensional image is numbered No. 4.

[0055] Case 2: Suppose that in a drone flight mission, there are a total of 4 drones, numbered UAV 1, UAV 2, UAV 3 and UAV 4. Correspondingly, the 3D point cloud image taken by the onboard radar of UAV No. 1 contains 3 drone targets numbered 2, 3 and 4, the 3D point cloud image taken by the onboard radar of UAV No. 2 contains 3 drone targets numbered 1, 3 and 4, the 3D point cloud image taken by the onboard radar of UAV No. 3 contains 3 drone targets numbered 1, 2 and 4, and the 3D point cloud image taken by the onboard radar of UAV No. 4 contains 3 drone targets numbered 1, 2 and 3.

[0056] Consider a 2D image captured by a camera in a multi-station visual intersection array. The image contains four drone targets to be numbered (numbered 1, 2, 3, and 4). Now, number the targets in this 2D image:

[0057] Take a 3D point cloud image (assuming it is a 3D point cloud image taken by the onboard radar of UAV No. 1, in which UAVs No. 2, No. 3 and No. 4 are present), and for UAV No. 2, calculate the degree of association between it and the four UAV targets in the two-dimensional image. After calculation, one target in the two-dimensional image is numbered 2. Replace the next numbered UAV target, namely UAV target No. 3, and calculate the degree of association between it and the four UAV targets in the two-dimensional image. After calculation, one target in the two-dimensional image is numbered 3; replace the next numbered UAV target, namely UAV target No. 4, and calculate the degree of association between it and the four UAV targets in the two-dimensional image. After calculation, one target in the two-dimensional image is numbered 4; now replace the next 3D point cloud image (assuming it is a 3D point cloud image taken by the onboard radar of UAV No. 2, in which UAVs No. 1, No. 3 and No. 4 are present), and calculate the degree of association between UAV No. 1 and the four UAV targets in the two-dimensional image. After calculation, one target in the two-dimensional image is numbered 1. At this point, all four UAV targets in this two-dimensional image are numbered.

[0058] Since the 3D point cloud image of the drone cluster at time t obtained by the drone carrying the airborne radar contains a total of M-1 drone targets, and the M-1 drone targets do not include the drone carrying the airborne radar, only any two 3D point cloud images are needed to cover all drone targets. Therefore, for each two-dimensional image, by comparing the target correlation degree with any 1-2 3D point cloud images in sequence, the drone number corresponding to each target in each two-dimensional image can be obtained.

Claims

1. A target matching method for a multi-station visual intersection array and a UAV cascade radar, characterized in that: The method comprises the following steps: S1, the multi-station visual intersection array uses N cameras to obtain N two-dimensional images of the UAV cluster at time t from different directions; S2. In the drone cluster, M drones are set up and numbered from drone 1 to drone M in sequence. Each drone carries an airborne radar to form a cascade radar. Each radar obtains a 3D point cloud image of the drone cluster at time t, and a total of M 3D point cloud images are obtained; S3, using the target recognition model to detect several drone targets and the two-dimensional detection frame corresponding to each drone target from each two-dimensional image; S4, using the point cloud target detection model to detect M-1 drone targets and the corresponding 3D detection box of each drone target from each 3D point cloud image; S5. For each two-dimensional image, compare the target association degree with any 1-2 3D point cloud images in sequence to obtain the drone number corresponding to each target in each two-dimensional image.

2. The target matching method of the multi-station visual intersection array and the UAV cascade radar according to claim 1 is characterized in that: N is greater than or equal to 3.

3. The target matching method of the multi-station visual intersection array and the UAV cascade radar according to claim 2 is characterized in that: The 3D point cloud image of the drone cluster at time t obtained by the drone carrying the airborne radar contains a total of M-1 drone targets, and each drone target carries a corresponding number. The drone carrying the airborne radar is not included in the M-1 drone targets.

4. The target matching method of the multi-station visual intersection array and the UAV cascade radar according to claim 3 is characterized in that: The target recognition model uses the YOLO series model.

5. The target matching method of the multi-station visual intersection array and the UAV cascade radar according to claim 4 is characterized in that: The point cloud target detection model uses the PointPillar point cloud target detection model.

6. The target matching method of the multi-station visual intersection array and the UAV cascade radar according to claim 5 is characterized in that: In step S5, assume that there are H drone targets in any two-dimensional image and perform the following operations: S61. Take a 3D point cloud image containing M-1 numbered drone targets; S62. For the first numbered UAV target, calculate the degree of correlation between it and the H UAV targets in the two-dimensional image respectively, and select the target with the highest correlation degree among the H UAV targets. The number of the UAV corresponding to this target is the same as the number corresponding to the first UAV target. If the degree of correlation between the first numbered UAV target and the H UAV targets in the two-dimensional image is less than 0.5, it is considered that there is no UAV target in the H UAV targets in the two-dimensional image with the same number as the first numbered UAV target. S63: Replace the next numbered drone target and repeat the operation in step S62 until all H drone targets in the two-dimensional image have been numbered. If there are still unnumbered drone targets in the two-dimensional image, proceed to step S64. S64. Take another 3D point cloud image, and take a drone target with a different number from that in the first 3D point cloud image. The number corresponding to this target is the number of the drone target that has not been numbered in the two-dimensional image.

7. The target matching method of the multi-station visual intersection array and the UAV cascade radar according to claim 6 is characterized in that: The degree of association between the drone target in the 3D point cloud image and the drone target in the 2D image is calculated as follows: extrinsic calibration is performed on the camera and airborne radar, the 3D detection frame in the 3D point cloud image is converted from the point cloud coordinate system to the image coordinate system, and the corresponding 2D point cloud detection frame is obtained. The intersection-and-union ratio between the 2D point cloud detection frame and the 2D detection frame of the drone target in the 2D image is calculated, which is the degree of association.

Images