Target matching method for multi-station visual intersection array and unmanned aerial vehicle cascade radar

Abstract

The invention discloses a target matching method of a multi-station visual intersection array and an unmanned aerial vehicle cascade radar. The method belongs to the technical field of target detection and matching. When a multi-station visual intersection array is used for unmanned aerial vehicle tracking, cameras at all angles can shoot a plurality of different unmanned aerial vehicle targets, at the moment, accurate matching of the targets shot by the cameras at all the angles becomes a technical problem, and the problem can be solved. The unmanned aerial vehicle targets in the multi-angle photos shot in the multi-station visual intersection array are accurately numbered through the determined serial numbers of the unmanned aerial vehicle targets in the cascade radar, so that in the subsequent processing process, which serial number each target in each angle photo in the multi-station visual intersection array specifically belongs to the unmanned aerial vehicle can be determined; therefore, accurate matching can be realized during triangulation positioning and image restoration of any target in the unmanned aerial vehicle cluster through a visual intersection array composed of multiple cameras, and target tracking errors are avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and matching, and particularly relates to the technical field of target matching between a multi-station vision intersection array and an unmanned aerial vehicle (UAV) cascaded radar. Background Art

[0002] With the rapid development of unmanned aerial vehicle (UAV) technology, UAV formations have shown great application potential in fields such as military reconnaissance, disaster monitoring, logistics transportation, agricultural plant protection, and geographical mapping. Efficient and accurate tracking of UAV formations is a key technical foundation for ensuring the safety of formation collaborative operations and realizing complex mission planning and control.

[0003] An optical camera (visible light / infrared) is used to image and track UAV targets. The vision system has a high angular resolution and can provide rich target appearance feature information (such as shape and texture), which is beneficial for target recognition. However, monocular vision is difficult to accurately obtain the three-dimensional position information of the target, and usually, a visual intersection array composed of multiple cameras is required for triangulation. However, when using a multi-station vision intersection array to track UAVs, each camera at each angle will capture several different UAV targets. At this time, the accurate matching of the targets captured by the cameras at each angle becomes a technical problem. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a target matching method between a multi-station vision intersection array and a UAV cascaded radar, which helps to accurately match the targets captured by the cameras at each angle in the multi-station vision intersection array through radar monitoring.

[0005] The method includes the following steps: S1. The multi-station vision intersection array uses N cameras to obtain N two-dimensional images of the UAV cluster at time t from different orientations; S2. In the UAV cluster, M UAVs are set, numbered from UAV No. 1 to UAV No. M in sequence. Each UAV carries an on-board radar to form a cascaded radar. Each radar obtains a 3D point cloud image of the UAV cluster at time t, and a total of M 3D point cloud images are obtained; S3. A target recognition model is used to detect several UAV targets and the corresponding two-dimensional detection frames of each UAV target from each two-dimensional image; S4. A point cloud target detection model is used to detect M - 1 UAV targets and the corresponding three-dimensional detection frames of each UAV target from each 3D point cloud image; S5. For each two-dimensional image, compare the degree of target association with any 1-2 three-dimensional point cloud images in sequence to obtain the UAV number corresponding to each target in each two-dimensional image.

[0006] Further, N is greater than or equal to 3.

[0007] Further, in the three-dimensional point cloud image of the UAV cluster at time t obtained by the UAV carrying the airborne radar, there are a total of M-1 UAV targets, and each UAV target carries a corresponding number. Among the M-1 UAV targets, this UAV carrying the airborne radar is not included.

[0008] Further, the target recognition model uses the YOLO series model.

[0009] Further, the point cloud target detection model uses the PointPillar point cloud target detection model.

[0010] Further, in step S5, let there be H UAV targets in any two-dimensional image, and perform the following operations: S61. Take a three-dimensional point cloud image, which contains M-1 numbered UAV targets; S62. For the first numbered UAV target, calculate the degree of association between it and the H UAV targets in the two-dimensional image respectively, and take the target with the highest degree of association among the H UAV targets. The number of the UAV corresponding to this target is the same as the number corresponding to the first numbered UAV target; if the degree of association between the first numbered UAV target and the H UAV targets in the two-dimensional image is less than 0.5, it is considered that there is no number in the H UAV targets in the two-dimensional image that is the same as the first numbered UAV target; S63. Replace with the next numbered UAV target and repeat the operation in step S62 until the numbers of all H UAV targets in the two-dimensional image have been determined. If there are still unnumbered UAV targets in the two-dimensional image, go to step S64; S64. Take another three-dimensional point cloud image and take the UAV target with a number different from that of the first three-dimensional point cloud image. The number corresponding to this target is the number of the unnumbered UAV target in the two-dimensional image. Further, calculating the degree of association between the UAV target in the three-dimensional point cloud image and the UAV target in the two-dimensional image specifically is: perform external parameter calibration on the camera and the airborne radar, convert the three-dimensional detection frame in the three-dimensional point cloud image from the point cloud coordinate system to the image coordinate system to obtain the corresponding two-dimensional point cloud detection frame, and calculate the intersection over union between the two-dimensional point cloud detection frame and the two-dimensional detection frame of the UAV target in the two-dimensional image, which is the degree of association.

[0011] The beneficial effects of the method of the present invention are as follows: By determining the numbers of the UAV targets in the cascaded radar, the UAV targets in the multi-angle photos taken by the multi-station vision intersection array are accurately numbered. In this way, in the subsequent processing process, it can be determined which numbered UAV each target in each angle photo of the multi-station vision intersection array specifically is, so that when subsequently performing triangulation positioning and image restoration on any target in the UAV cluster through the vision intersection array composed of multiple cameras, accurate matching can be achieved, avoiding target tracking errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of the method described in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0014] Embodiment 1 This embodiment provides a method for target matching between a multi-station vision intersection array and a UAV cascaded radar. By determining the numbers of the UAV targets in the cascaded radar, the UAV targets in the multi-angle photos taken by the multi-station vision intersection array are accurately numbered. In this way, in the subsequent processing process, it can be determined which numbered UAV each target in each angle photo of the multi-station vision intersection array specifically is, so that when subsequently performing triangulation positioning and image restoration on any target in the UAV cluster through the vision intersection array composed of multiple cameras, accurate matching can be achieved, avoiding target tracking errors.

[0015] As Figure 1 shown, the method of the present invention includes the following steps: S1. The multi-station vision intersection array uses N cameras to obtain N two-dimensional images of the UAV cluster at time t from different orientations; S2. In the UAV cluster, M UAVs are set, numbered from UAV No. 1 to UAV No. M in sequence. Each UAV is equipped with an on-board radar to form a cascaded radar. Each radar obtains a 3D point cloud image of the UAV cluster at time t, and a total of M 3D point cloud images are obtained; S3. Use a target recognition model to detect several UAV targets and the corresponding two-dimensional detection frames for each UAV target from each two-dimensional image; S4. Use a point cloud target detection model to detect M - 1 UAV targets and the corresponding three-dimensional detection frames for each UAV target from each 3D point cloud image; S5. For each two-dimensional image, compare the degree of target association with any 1-2 three-dimensional point cloud images in sequence to obtain the UAV number corresponding to each target in each two-dimensional image.

[0016] Multi-station vision intersection arrays usually use cameras with at least 3 perspectives for shooting from different angles, so N is greater than or equal to 3.

[0017] In the three-dimensional point cloud image of the UAV cluster at time t obtained by the UAV carrying the airborne radar, there are a total of M-1 UAV targets, and each UAV target carries a corresponding number. Among the M-1 UAV targets, this UAV carrying the airborne radar is not included.

[0018] The target recognition model uses the YOLO series of models. After verification, YOLOv11 can be used as the optimal choice.

[0019] The point cloud target detection model uses the PointPillar point cloud target detection model.

[0020] In step S5, let there be H UAV targets in any two-dimensional image, and perform the following operations: S51. Take a three-dimensional point cloud image, which contains M-1 numbered UAV targets; S52. For the first numbered UAV target, calculate the degree of association between it and the H UAV targets in the two-dimensional image respectively, and take the target with the highest degree of association among the H UAV targets. The number of the UAV corresponding to this target is the same as the number corresponding to the first UAV target; if the degree of association between the first numbered UAV target and the H UAV targets in the two-dimensional image is less than 0.5, it is considered that there is no number in the H UAV targets in the two-dimensional image that is the same as the first numbered UAV target; S53. Replace it with the next numbered UAV target and repeat the operation in step S52 until the numbers of all H UAV targets in the two-dimensional image have been determined. If there are still unnumbered UAV targets in the two-dimensional image, go to step S54; S54. Take another three-dimensional point cloud image and take the UAV target with a different number from the first three-dimensional point cloud image. The number corresponding to this target is the number of the unnumbered UAV target in the two-dimensional image. The specific calculation of the degree of association between the UAV target in the three-dimensional point cloud image and the UAV target in the two-dimensional image is as follows: Calibrate the external parameters of the camera and the airborne radar, convert the three-dimensional detection box in the three-dimensional point cloud image from the point cloud coordinate system to the image coordinate system to obtain the corresponding two-dimensional point cloud detection box, and calculate the intersection over union between the two-dimensional point cloud detection box and the two-dimensional detection box of the UAV target in the two-dimensional image, which is the degree of association.

[0021] Example 2 This example further limits Example 1. The joint calibration of the camera and the lidar is to obtain the conversion relationship between the camera coordinate system and the point cloud coordinate system, that is, the rotation and translation relationship between the two three-dimensional coordinate systems. Denote the external parameters between the two sensors as , , where is the rotation matrix, is the translation matrix, and the formula is as follows: ; Among them, is the point coordinate in the point cloud coordinate system, is the point coordinate in the camera coordinate system. The conversion relationship from the point cloud coordinate system to the pixel coordinate system is as follows: ; Furthermore, it can be obtained that: ; Among them, is the pixel point corresponding to the pixel coordinate system corresponding to the camera, is the origin of the pixel coordinate system corresponding to the camera, , , is the camera focal length, and are the number of pixels in the row and column directions of the image coordinate system corresponding to the camera; , and are all 1×4 transformation matrices. Let be the augmented matrix, which has 12 unknowns. Therefore, at least six pairs of matching points are required during the calibration process to linearly solve the augmented matrix .

[0022] Using the above method, the three-dimensional detection frame in the 3D point cloud image is converted from the point cloud coordinate system to the image coordinate system, and the intersection over union (IoU) between the two-dimensional point cloud detection frame and the two-dimensional detection frame of the UAV target in the two-dimensional image is calculated. The IoU represents the matching degree of the two rectangular detection frames. The better the matching degree, the higher the correlation degree. According to the correlation degree, it is judged whether the two target frames point to the same object.

[0023] Example 3 This example further limits Example 1 and further limits step S5.

[0024] Case 1: Suppose in a drone flight mission, there are 4 drones, numbered as Drone No. 1, Drone No. 2, Drone No. 3, and Drone No. 4 respectively. Correspondingly, the 3D point cloud image captured by the on-board radar of Drone No. 1 contains 3 drone targets numbered 2, 3, and 4; the 3D point cloud image captured by the on-board radar of Drone No. 2 contains 3 drone targets numbered 1, 3, and 4; the 3D point cloud image captured by the on-board radar of Drone No. 3 contains 3 drone targets numbered 1, 2, and 4; the 3D point cloud image captured by the on-board radar of Drone No. 4 contains 3 drone targets numbered 1, 2, and 3.

[0025] There is a two-dimensional image captured by a camera in an existing multi-station vision intersection array, in which there are 2 drone targets to be numbered (assumed to be Drone No. 3 and Drone No. 4). Now, number the targets in this two-dimensional image: Take a 3D point cloud image (assumed to be the 3D point cloud image captured by the on-board radar of Drone No. 1, in which there are Drones No. 2, No. 3, and No. 4). For Drone No. 2, calculate the degree of association between it and the 2 drone targets in the two-dimensional image respectively. The degrees of association are both less than 0.5. At this time, it is considered that there is no number in the 2 drone targets in the two-dimensional image that is the same as Drone No. 2; Replace it with the next numbered drone target, i.e., Drone No. 3 target, and calculate the degree of association between it and the 2 drone targets in the two-dimensional image respectively. After calculation, one of the targets in the two-dimensional image is numbered as 3; Replace it with the next numbered drone target, i.e., Drone No. 4 target, and calculate the degree of association between it and the 2 drone targets in the two-dimensional image respectively. After calculation, one of the targets in the two-dimensional image is numbered as 4.

[0026] Case 2: Suppose in a drone flight mission, there are 4 drones, numbered as Drone No. 1, Drone No. 2, Drone No. 3, and Drone No. 4 respectively. Correspondingly, the 3D point cloud image captured by the on-board radar of Drone No. 1 contains 3 drone targets numbered 2, 3, and 4; the 3D point cloud image captured by the on-board radar of Drone No. 2 contains 3 drone targets numbered 1, 3, and 4; the 3D point cloud image captured by the on-board radar of Drone No. 3 contains 3 drone targets numbered 1, 2, and 4; the 3D point cloud image captured by the on-board radar of Drone No. 4 contains 3 drone targets numbered 1, 2, and 3.

[0027] There is a two-dimensional image captured by a camera in an existing multi-station vision intersection array, in which there are 4 drone targets to be numbered (in sequence as No. 1, No. 2, No. 3, and No. 4). Now, number the targets in this two-dimensional image: Take a 3D point cloud image (assume it is a 3D point cloud image captured by the on-board radar of UAV No. 1, in which there are UAV No. 2, UAV No. 3 and UAV No. 4). For UAV No. 2 among them, calculate the degree of association between it and the 4 UAV targets in the 2D image respectively. After calculation, one target in the 2D image is numbered as No. 2. Then replace it with the next numbered UAV target, i.e., UAV No. 3 target, and calculate the degree of association between it and the 4 UAV targets in the 2D image respectively. After calculation, one target in the 2D image is numbered as No. 3; then replace it with the next numbered UAV target, i.e., UAV No. 4 target, and calculate the degree of association between it and the 4 UAV targets in the 2D image respectively. After calculation, one target in the 2D image is numbered as No. 4; at this time, replace it with the next 3D point cloud image (assume it is a 3D point cloud image captured by the on-board radar of UAV No. 2, in which there are UAV No. 1, UAV No. 3 and UAV No. 4). For UAV No. 1 among them, calculate the degree of association between it and the 4 UAV targets in the 2D image respectively. After calculation, one target in the 2D image is numbered as No. 1. Thus, the 4 UAV targets in this 2D image are all numbered.

[0028] Since the 3D point cloud image of the UAV cluster at time t obtained by the UAV carrying the on-board radar contains a total of M - 1 UAV targets, and among the M - 1 UAV targets, this UAV carrying the on-board radar is not included, so only any two 3D point cloud images are needed to cover all UAV targets. Therefore, for each 2D image, by comparing the degree of target association with any 1 - 2 3D point cloud images in turn, the UAV number corresponding to each target in each 2D image can be obtained.

Claims

1. A target matching method for a multi-station vision intersection array and an unmanned aerial vehicle cascaded radar, characterized in that The method includes the following steps: S1. The multi-station vision intersection array uses N cameras to obtain N two-dimensional images of the UAV cluster at time t from different orientations; S2. In the UAV cluster, M UAVs are set and numbered from UAV No. 1 to UAV No. M in sequence. Each UAV carries an on-board radar to form a cascaded radar. Each radar obtains a 3D point cloud image of the UAV cluster at time t, and a total of M 3D point cloud images are obtained; S3. The target recognition model is used to detect several UAV targets and the corresponding two-dimensional detection frames for each UAV target from each two-dimensional image; S4. The point cloud target detection model is used to detect M - 1 UAV targets and the corresponding three-dimensional detection frames for each UAV target from each 3D point cloud image; S5. For each two-dimensional image, by comparing the target association degree with any 1 - 2 3D point cloud images in sequence, the UAV number corresponding to each target in each two-dimensional image is obtained.

2. The method for target matching of a multi-station vision intersection array and an unmanned aerial vehicle cascaded radar according to claim 1, wherein N is greater than or equal to 3.

3. The target matching method of the multi-station vision intersection array and the UAV cascaded radar according to claim 2, characterized in that, In the 3D point cloud image of the UAV cluster obtained by the UAV carrying the on-board radar at time t, there are a total of M - 1 UAV targets, and each UAV target carries a corresponding number. Among the M - 1 UAV targets, this UAV carrying the on-board radar is not included.

4. The method for target matching of the multi-station vision intersection array and the UAV cascaded radar according to claim 3, characterized in that The target recognition model uses the YOLO series model.

5. The target matching method of the multi-station vision intersection array and the UAV cascaded radar according to claim 4, characterized in that, The point cloud target detection model uses the PointPillar point cloud target detection model.

6. The target matching method of the multi-station vision intersection array and the UAV cascaded radar according to claim 5, characterized in that In step S5, let there be H UAV targets in any two-dimensional image, and perform the following operations: S61. Take a 3D point cloud image, which contains M - 1 numbered UAV targets; S62. For the first numbered UAV target, calculate the association degree between it and the H UAV targets in the two-dimensional image respectively, and take the target with the highest association degree among the H UAV targets. The number of the UAV corresponding to this target is the same as the number corresponding to the first numbered UAV target; if the association degree between the first numbered UAV target and the H UAV targets in the two-dimensional image is less than 0.5, it is considered that there is no number in the H UAV targets in the two-dimensional image that is the same as the first numbered UAV target; S63. Replace it with the next numbered UAV target and repeat the operation in step S62 until the numbers of all H UAV targets in the two-dimensional image have been determined. If there are still unnumbered UAV targets in the two-dimensional image, go to step S64; S64. Take another 3D point cloud image, and take the UAV target with a number different from that of the first 3D point cloud image. The number corresponding to this target is the number of the unnumbered UAV target in the two-dimensional image.

7. The method for target matching of the multi-station vision intersection array and the UAV cascaded radar according to claim 6, wherein Calculating the association degree between the UAV target in the 3D point cloud image and the UAV target in the two-dimensional image specifically is: calibrate the external parameters of the camera and the on-board radar, transform the three-dimensional detection frame in the 3D point cloud image from the point cloud coordinate system to the image coordinate system to obtain the corresponding two-dimensional point cloud detection frame, and calculate the intersection over union between the two-dimensional point cloud detection frame and the two-dimensional detection frame of the UAV target in the two-dimensional image, which is the association degree.

Images