An unmanned aerial vehicle target cross-phase association method based on panoramic vision

Abstract

The application belongs to the technical field of image recognition, and discloses a UAV target cross-camera association method based on panoramic vision, which comprises the following steps: S1, panoramic vision camera arrangement and camera internal and external parameter calibration; S2, image distortion removal and determination of overlapping area between cameras: removing the distortion of the original images of the 8 cameras by using distortion coefficients, and determining the image overlapping width between the two adjacent cameras; S3, UAV recognition data set construction and model training: constructing a UAV sample set containing a rotating labeled box and a category label, constructing a YOLO11n-obb convolutional neural network and training; S5, constructing a Hungarian cost matrix between the targets of adjacent cameras, and performing optimal pairing; S6, repeatedly performing steps S4-S5 on all adjacent cameras to generate all UAV target position information after eliminating cross-camera repetition. The application reduces the problems of inaccurate cross-camera target matching caused by small target detection boxes and unobvious details in the image coordinate system.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and in particular relates to a method for cross-camera association of UAV targets based on panoramic vision. Background Technology

[0002] When multiple cameras detect targets, overlapping areas between cameras can cause the same target to be detected twice, making it impossible to manage target IDs uniformly across multiple cameras.

[0003] To associate targets between two adjacent cameras, matching is typically performed based on the feature similarity between image detection boxes. Chinese patent "CN202510788250.4 Cross-Camera Vehicle Matching Method, Apparatus, Device, Medium, and Program Product" provides a method for matching targets based on detailed features between cameras. This method extracts detailed features from the target vehicle and the set of secondary candidate vehicles using a target detail matching model, obtaining detailed target vehicle features and detailed secondary vehicle features. Secondary matching is then performed on the detailed target vehicle features to obtain the matching result. However, because vehicle targets are large and their detailed information is obvious, this method is easy to use for association and matching. But for small targets like drones, which are far away and lack obvious image detail information, it is difficult to associate targets using image matching methods. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for cross-camera association of UAV targets, so as to ensure accurate target matching in the overlapping area of ​​multiple cameras.

[0005] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows: A method for cross-camera target association in UAVs based on panoramic vision includes the following steps: S1. Deployment of panoramic vision cameras and calibration of camera intrinsic and extrinsic parameters: Eight visible light cameras are evenly distributed and installed with the center of the roof as the origin to ensure that the eight cameras can cover a 360° panoramic view, and there is an overlapping area between the fields of view of adjacent cameras. Then, the internal and external parameters of the eight visible light cameras that make up the panoramic vision camera are calibrated respectively. S2. Image distortion correction, determining overlapping areas between cameras: Distortion coefficients are used to distort the original images from the 8 cameras to determine the image overlap width between adjacent cameras. S3. Construction of drone identification dataset and model training: A UAV sample set containing rotated bounding boxes and category labels is constructed, and a YOLO11n-obb convolutional neural network is constructed and trained to obtain a UAV target recognition model. S4. Monocular calculation of the three-dimensional coordinates of the UAV roof center coordinate system: The trained YOLO11n-obb model was used to detect drones in eight images, retaining only the rotated detection boxes whose center points fell within the overlapping area. For the effective detection boxes of adjacent cameras A and B, the top-left and bottom-left pixels of the rotated boxes were extracted to calculate the drone pixel height h. The drone's true physical height H was matched according to the detection category. The target camera depth Zc was solved by combining the camera intrinsic parameters and similar triangles through pinhole imaging. Then, the three-dimensional coordinates (Xc, Yc, Zc) in the target camera coordinate system were calculated based on the center pixel coordinates of the detection boxes. The camera coordinates were transformed to the roof center coordinate system using the extrinsic parameters of each camera, and the three-dimensional coordinate sets of the drone roof coordinate system corresponding to cameras A and B were obtained respectively. S5. Construct the Hungarian cost matrix between adjacent camera targets and perform optimal pairing. A L2 distance cost matrix is ​​constructed using the target's 3D coordinates in the vehicle-top coordinate system of cameras A and B. The Hungarian algorithm is used to achieve one-to-one optimal target matching. For the same UAV that is successfully matched, the average of the two sets of 3D coordinates is used as the fused target coordinates for storage. All unmatched targets are stored independently. S6. Repeat steps S4 to S5 for all adjacent cameras to generate all UAV target position information after eliminating cross-camera duplication.

[0006] Furthermore, in step S1, the camera intrinsic parameters are obtained using the Zhang Zhengyou calibration method. K n and distortion coefficient dist n : Where n represents the camera number, the intrinsic parameters of the 8 cameras are K1, K2…K8, and the distortion coefficients are dist1, dist2…dist8; K n f in x f represents the focal length in the x-direction. y c represents the focal length in the y-direction. x c represents the pixel coordinates in the x-direction of the optical center. y Dist represents the pixel coordinates in the y-direction of the optical center; n In this context, k1 represents first-order radial distortion, k2 represents second-order radial distortion, k3 represents third-order radial distortion, p1 represents vertical tangential offset, and p2 represents horizontal tangential offset. Camera extrinsic calibration: Generate the rotation matrix R from the camera coordinate system to the roof center point coordinate system. n Translation matrix T n The rotation matrices for the 8 cameras are R1, R2…R8, and the translation matrices are T1, T2…T8.

[0007] Furthermore, in step S3, after constructing the YOLO11n-obb convolutional neural network, N drone categories are set, where N is the number of common drone categories (more than 500 manually labeled samples for each drone category), and the YOLO11n-obb convolutional neural network is trained until the network converges to obtain the drone target recognition model.

[0008] Furthermore, in step S4, the rotation detection box in the overlapping area of ​​camera A is... Category label is , where i represents the number of detection boxes for camera A; The rotation detection box in the overlapping area of ​​camera B is: Category label is j represents the number of detection boxes for camera B; Take the top-left corner (x1, y1) and bottom-left corner (x2, y2) of the rotation detection frame of camera A. The pixel distance between the two points, which is the pixel height h of the drone, is: The true altitude H of the drone is obtained based on the category labels provided by the model. The target camera depth Z is obtained based on the similar triangle formula and camera intrinsic parameters. c for: The center point of the rotating detection frame is (x0, y0). The X and Y coordinates of the UAV in the camera coordinate system are calculated using the following formula. c and Y c They are respectively: Therefore, the coordinates of the rotated detection box after transformation to the camera coordinate system are (X... c Y c Z c ), rotate the detection box in the overlapping area of ​​camera A. Transform to camera coordinate system Then, based on the extrinsic parameters of camera A, the camera coordinates are transformed to the coordinate system of the roof center. Below, represented as (X) v Y v Z v ),get The coordinates of the target vehicle roof center in the overlapping area of ​​camera B are obtained in the same way. .

[0009] Furthermore, in step S5, the drone target is used with camera A. Coordinates and B-camera drone target Coordinates, Construction and The cost matrix for one-to-one Hungarian matching between them: Given the cost of a one-to-one match between targets in camera A and camera B, find the one-to-one target pair that minimizes the total cost.

[0010] This invention has the following advantages: It employs a YOLOv11n-obb rotating bounding box to capture UAV targets, identifies the UAV category, and obtains the target's prior altitude. Then, it uses a pinhole camera algorithm to calculate the target distance based on camera intrinsic parameters, improving the accuracy of the target distance calculation. Based on the calculated target distance, the target pixel coordinates are transformed to camera coordinates, and then further transformed to the vehicle roof center coordinate system based on camera extrinsic parameters, thus obtaining the target position information in the vehicle roof center coordinate system for all cameras. A Hungarian matching algorithm is used to pair UAV targets in overlapping areas one-to-one; paired targets are fused, while unpaired targets are all retained. This reduces problems such as inaccurate cross-camera target matching caused by small target detection boxes and unclear details in the image coordinate system. Attached Figure Description

[0011] Figure 1 This is a flowchart of the UAV target cross-camera association method based on panoramic vision according to the present invention. Detailed Implementation

[0012] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0013] like Figure 1 As shown, the present invention provides a method for cross-camera association of UAV targets based on panoramic vision, comprising the following steps: (1) Installation of panoramic vision cameras and calibration of camera intrinsic and extrinsic parameters. Eight panoramic vision cameras are evenly distributed and installed with the center of the vehicle roof as the origin, ensuring that the eight cameras can cover 360° around the perimeter and that there is an overlapping area between the cameras. Then, the intrinsic and extrinsic parameters of each camera are calibrated. In this embodiment, the intrinsic and extrinsic parameters of the panoramic vision camera composed of eight visible light cameras need to be calibrated separately.

[0014] The camera intrinsic parameters were obtained using Zhang Zhengyou's calibration method. K n and distortion coefficient dist n : , Where n represents the number of cameras, the intrinsic parameters of the 8 cameras are K1, K2…K8, and the distortion coefficients are dist1, dist2…dist8. K n f in x f represents the focal length in the x-direction. y c represents the focal length in the y-direction. x c represents the pixel coordinates in the x-direction of the optical center. y This represents the pixel coordinates in the y-direction of the optical center. (dist) n In the diagram, k1 represents first-order radial distortion, k2 represents second-order radial distortion, k3 represents third-order radial distortion, p1 represents vertical tangential offset, and p2 represents horizontal tangential offset.

[0015] Camera extrinsic parameter calibration generates rotation (R) and translation (T) matrices from the camera coordinate system (origin at the camera optical center, X-axis to the right, Y-axis down, Z-axis forward) to the vehicle roof center coordinate system (origin at the vehicle roof center, X-axis forward, Y-axis left, Z-axis up). The rotation matrices for the 8 cameras are R1, R2…R8, and the translation matrices are T1, T2…T8.

[0016] (2) Distortion coefficients are used to distort the original images of the 8 cameras to obtain distorted images with a resolution of W×H, where W is the image width and H is the image height. The image overlap width D between adjacent cameras is determined. Therefore, the image width range of camera 1 is (WD)~W, which overlaps with the image width range of camera 2 (0~D). The overlap areas between other adjacent cameras are deduced in the same way.

[0017] (3) Construct the UAV target sample set S drone UAV target sample set S drone Images of various common drone targets are collected and manually labeled to obtain rotated bounding boxes (Groundtruth) and drone categories. These Groundtruths and drone categories are used as training labels to construct a YOLO11n-obb convolutional neural network. N drone categories are set, where N is the number of common drone categories. X images (typically 500 or more) are manually labeled for each drone category. The YOLO11n-obb convolutional neural network is trained until it converges, resulting in the drone target recognition model Net. drone Net drone It can detect and identify drone targets in images and provide corresponding drone category labels. , representing the categories of M drone targets in the image, and their location information in the image. , represents the bounding rectangle of the M drone targets in the image.

[0018] (4) Use the YOLO11n-obb model to perform UAV recognition on the 8 images respectively, and filter the detection boxes whose center points are in the overlapping area. Assuming that camera A and camera B are adjacent, the rotation detection box in the overlapping area of ​​camera A is: Category label is 'i' represents the number of detection boxes in camera A. The rotated detection boxes within the overlapping region of camera B are... Category label is , j represents the number of detection boxes for camera B. Taking the top-left corner (x1, y1) and bottom-left corner (x2, y2) of the rotating detection box for camera A, the pixel distance between these two points, which is the pixel height h of the drone, is: The true altitude H of the drone is obtained based on the category labels provided by the model. The distance Z of the drone is obtained using the formula for similar triangles and the camera intrinsic parameters K. c for: The center point of the rotating detection frame is (x0, y0). The X and Y coordinates of the UAV in the camera coordinate system are calculated using the following formula. c and Y c They are respectively: Therefore, the rotating detection box is transformed into the camera coordinate system. The coordinates after the bottom are (X) c Y c Z c ), rotate the detection box in the overlapping area of ​​camera A. Transform to camera coordinate system Then, based on the extrinsic parameters (R and T matrices) of camera A, the camera coordinates are transformed to the coordinate system of the vehicle roof center. Below, represented as (X) v Y v Z v ),get The coordinates of the target vehicle roof center in the overlapping area of ​​camera B are obtained in the same way. .

[0019] (5) Using camera A to target the drone Coordinates and B-camera drone target Coordinates, Construction and The cost matrix for one-to-one Hungarian matching between them: Given the cost of a one-to-one match between targets in camera A and camera B, find the one-to-one target pair that minimizes the total cost. For a matched target, such as the target under camera A... ( ) and B camera , ( () represents the coordinates under camera A. Let B represent the coordinates under camera B. Then, for target fusion, the average of the two coordinates is calculated and saved as the new target position information. For targets that are not matched by cameras A and B, they are saved as new targets.

[0020] (6) Perform the above operations on all 8 cameras to generate all UAV target location information after cross-camera target fusion.

[0021] Compared with the prior art, the technical solution proposed in this invention establishes a cross-camera matching technology in the target space coordinate system of a panoramic camera UAV, which reduces the problems of inaccurate cross-camera target matching caused by small target detection boxes and unclear details in the image coordinate system.

[0022] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art will be able to make various modifications and improvements without departing from the principles of the present invention, and these modifications and improvements should also be considered to fall within the scope of protection of the present invention.

Claims

1. A method for cross-camera association of UAV targets based on panoramic vision, characterized in that, Includes the following steps: S1. Deployment of panoramic vision cameras and calibration of camera intrinsic and extrinsic parameters: Eight visible light cameras are evenly distributed and installed with the center of the roof as the origin to ensure that the eight cameras can cover a 360° panoramic view, and there is an overlapping area between the fields of view of adjacent cameras. Then, the internal and external parameters of the eight visible light cameras that make up the panoramic vision camera are calibrated respectively. S2. Image distortion correction, determining overlapping areas between cameras: Distortion coefficients are used to distort the original images from the 8 cameras to determine the image overlap width between adjacent cameras. S3. Construction of drone identification dataset and model training: A UAV sample set containing rotated bounding boxes and category labels is constructed, and a YOLO11n-obb convolutional neural network is constructed and trained to obtain a UAV target recognition model. S4. Monocular calculation of the three-dimensional coordinates of the UAV roof center coordinate system: The trained YOLO11n-obb model was used to detect drones in eight images, retaining only the rotated detection boxes whose center points fell within the overlapping area. For the effective detection boxes of adjacent cameras A and B, the top-left and bottom-left pixels of the rotated boxes were extracted to calculate the drone pixel height h. The drone's true physical height H was matched according to the detection category. The target camera depth Zc was solved by combining the camera intrinsic parameters and similar triangles through pinhole imaging. Then, the three-dimensional coordinates (Xc, Yc, Zc) in the target camera coordinate system were calculated based on the center pixel coordinates of the detection boxes. The camera coordinates were transformed to the roof center coordinate system using the extrinsic parameters of each camera, and the three-dimensional coordinate sets of the drone roof coordinate system corresponding to cameras A and B were obtained respectively. S5. Construct the Hungarian cost matrix between adjacent camera targets and perform optimal pairing: A L2 distance cost matrix is ​​constructed using the target's 3D coordinates in the vehicle-top coordinate system of cameras A and B. The Hungarian algorithm is used to achieve one-to-one optimal target matching. For the same UAV that is successfully matched, the average of the two sets of 3D coordinates is used as the fused target coordinates for storage. All unmatched targets are stored independently. S6. Repeat steps S4 to S5 for all adjacent cameras to generate all UAV target position information after eliminating cross-camera duplication.

2. The method for cross-camera association of UAV targets based on panoramic vision according to claim 1, characterized in that, In step S1, the camera intrinsic parameters are obtained using the Zhang Zhengyou calibration method. K n and distortion coefficient dist n : Where n represents the camera number, the intrinsic parameters of the 8 cameras are K1, K2…K8, and the distortion coefficients are dist1, dist2…dist8; K n f in x f represents the focal length in the x-direction. y c represents the focal length in the y-direction. x c represents the pixel coordinates in the x-direction of the optical center. y Dist represents the pixel coordinates in the y-direction of the optical center; n In this context, k1 represents first-order radial distortion, k2 represents second-order radial distortion, k3 represents third-order radial distortion, p1 represents vertical tangential offset, and p2 represents horizontal tangential offset. Camera extrinsic calibration: Generate the rotation matrix R from the camera coordinate system to the roof center point coordinate system. n Translation matrix T n The rotation matrices for the 8 cameras are R1, R2…R8, and the translation matrices are T1, T2…T8.

3. The method for cross-camera association of UAV targets based on panoramic vision according to claim 1, characterized in that, In step S3, after constructing the YOLO11n-obb convolutional neural network, N drone categories are set, where N is the number of common drone categories (more than 500 manually labeled samples for each drone category). The YOLO11n-obb convolutional neural network is trained until the network converges, thus obtaining the drone target recognition model.

4. The method for cross-camera association of UAV targets based on panoramic vision according to claim 1, characterized in that, In step S4, the rotation detection box in the overlapping area of ​​camera A is... Category label is , where i represents the number of detection boxes for camera A; The rotation detection box in the overlapping area of ​​camera B is: Category label is j represents the number of detection boxes for camera B; Take the top-left corner (x1, y1) and bottom-left corner (x2, y2) of the rotation detection frame of camera A. The pixel distance between the two points, which is the pixel height h of the drone, is: The true altitude H of the drone is obtained based on the category labels provided by the model. The target camera depth Z is obtained based on the similar triangle formula and camera intrinsic parameters. c for: The center point of the rotating detection frame is (x0, y0). The X and Y coordinates of the UAV in the camera coordinate system are calculated using the following formula. c and Y c They are respectively: Therefore, the coordinates of the rotated detection box after transformation to the camera coordinate system are (X... c Y c Z c ), rotate the detection box in the overlapping area of ​​camera A. Transform to camera coordinate system Then, based on the extrinsic parameters of camera A, the camera coordinates are transformed to the coordinate system of the roof center. Below, represented as (X) v Y v Z v ),get The coordinates of the target vehicle roof center in the overlapping area of ​​camera B are obtained in the same way. .

5. The method for cross-camera association of UAV targets based on panoramic vision according to claim 1, characterized in that, In step S5, the drone target is used with camera A. Coordinates and B-camera drone target Coordinates, Construction and The cost matrix for one-to-one Hungarian matching between them: Given the cost of a one-to-one match between targets in camera A and camera B, find the one-to-one target pair that minimizes the total cost.

Images