A camera self-calibration and target tracking method based on multi-mode information reference in a large-scale scene of an airport

Abstract

The application designs a camera self-calibration and target tracking method based on multi-mode information reference in a large airport scene, and belongs to the technical field of computer vision. The application obtains three-level coordinates of an airplane geographic coordinate, actual pixel coordinate and theoretical pixel coordinate through airplane navigation information and camera related parameters, then solves the corresponding matrix of the pixel coordinate and the geographic coordinate according to the geometric characteristics of the geographic coordinate and the corresponding relationship among the three-level coordinates, introduces a typical reference point for external parameter calibration, obtains a detection error through the real-time position of a typical target point solved by the internal and external parameters of the camera, and finally solves the real-time position of a non-cooperative detection target by using the corresponding matrix and combining a target detection algorithm, which has great significance in the field of safety monitoring in a large airport scene.

Description

Technical Field

[0001] This invention relates to the field of camera calibration technology and proposes a camera self-calibration and target tracking method based on multi-modal information reference in large-scale airport scenarios, which is of great significance for security monitoring in large-scale airport scenarios. Background Technology

[0002] In the field of machine vision, camera calibration is a prerequisite and key technology for acquiring ideal visual data. However, technical issues such as manufacturing processes can lead to camera errors, and during use, various physical or chemical factors can cause the camera to continuously generate new errors, which can even accumulate. Therefore, in practice, camera calibration is necessary, and the calibration results must be continuously corrected to achieve the desired performance.

[0003] According to existing camera calibration techniques, they can be broadly classified into three categories: traditional photogrammetry camera calibration methods, active vision calibration methods, and self-calibration methods. Traditional photogrammetry camera calibration methods require precisely manufactured calibration blocks. Calibration is completed by measuring the positional relationship between the camera and the calibration block, based on the known geometric configuration of the calibration block. Active vision methods require controlling the camera to perform precise, specific movements, such as rotation around the optical center or pure translation, and then completing calibration based on images captured at multiple locations and the camera's attitude at each location. Self-calibration techniques utilize the camera's own constraints, taking multiple shots of the same stationary object or simultaneously shooting with multiple cameras, and using the constraints between image sequences for calibration. This method has poor accuracy and robustness, and is not suitable for real-time applications.

[0004] In large-scale airport scenarios, traditional photogrammetry camera calibration methods struggle to meet requirements in terms of calibration block size, quantity, and manufacturing precision, resulting in high costs and low calibration accuracy. For active vision calibration methods, most existing airport monitoring equipment does not support controlled, precise motion, necessitating modifications that are extremely costly. Simple camera self-calibration techniques suffer from low calibration accuracy and robustness, making direct application difficult.

[0005] In summary, existing research lacks a calibration and target tracking method applicable to large-scale airport scenarios. This invention proposes a method based on the correspondence between multi-level aircraft coordinates, using a set of points with typical geometric constraints within the airport for camera extrinsic parameter calibration, and combining this with a target detection algorithm to calculate target position information in real time. Summary of the Invention

[0006] The purpose of this invention is to propose a camera self-calibration and target tracking method based on multi-modal information reference in a large-scale airport scene.

[0007] The present invention is characterized by providing a camera self-calibration method based on multi-mode information reference in a large-scale airport scene. Combining the characteristic that the relative motion between the aircraft and the camera can be accurately measured in a large-scale airport scene, and relying on multi-mode information composed of aircraft navigation information and typical airport geometric constraints, the method is based on the camera self-calibration method and integrates the characteristics of traditional camera calibration methods in photogrammetry and active visual calibration methods. This method solves the problems of poor accuracy and timeliness of camera self-calibration in a large-scale airport scene.

[0008] The camera self-calibration and target tracking method based on multi-modal information reference in a large-scale airport scene provided by this invention includes the following steps:

[0009] 1) Obtain the aircraft's geographic coordinates through the GBAS system;

[0010] 2) Detect the aircraft pixel coordinates within the image and calculate the ideal coordinates of the aircraft in the pixel coordinate system;

[0011] 3) Form a matrix corresponding to the aircraft coordinates at the image pixel level and the geospatial level;

[0012] 4) Find typical target points as reference points and determine typical geometric constraints;

[0013] 5) Based on the geometric constraints determined by the reference point and the multi-mode information composed of the aircraft's multi-level coordinates, perform camera extrinsic parameter calibration;

[0014] 6) Real-time positions of typical target points calculated using the corresponding matrix;

[0015] 7) Based on requirements, further calculate the real-time location of non-cooperative detection targets;

[0016] The system of the present invention has the following characteristics:

[0017] 1. The algorithm is simple and highly versatile.

[0018] The method of the present invention has the following characteristics:

[0019] 1. Effectively and continuously correct the calibration results to achieve the desired usage effect;

[0020] 2. Perform camera calibration by combining the camera's physical parameters with the aircraft's coordinates at various levels;

[0021] 3. It can calculate the location information of targets in a large-scale scene in real time. Attached Figure Description

[0022] Figure 1 Workflow diagram of camera self-calibration and target tracking method based on multi-modal information reference in large-scale airport scenarios

[0023] Figure 2 Ideal pixel coordinate calculation flowchart Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention proposes a camera self-calibration and target tracking method based on geometric constraints and navigation information in a large-scale airport scenario. This method can effectively improve calibration accuracy and real-time performance, and is of great significance for safety monitoring in a large-scale airport scenario.

[0026] The specific implementation process in practical applications is as follows: Figure 1 As shown, specifically:

[0027] 1) Using the GBAS system, the geographical location information of the aircraft traveling along the guide line on the ground under large-scale monitoring is sampled at appropriate time intervals to obtain the corresponding t. k Time of flight geographic coordinates W k (x k ,y k ,z k ), k=1, 2, 3, 4,...,n;

[0028] 2) Determine the relative positional relationship between geographic information and the aircraft body. Using the target detection algorithm, based on the geometric dimensions of the aircraft, obtain the pixel coordinates of the aircraft at the corresponding time in the image. Based on the actual coordinates of the aircraft and various parameters of the camera, use the pinhole imaging model to calculate the ideal coordinates of the aircraft in the pixel coordinate system at each time.

[0029] First, determine the geographic coordinates V. k Detailed location on the fuselage;

[0030] Then, using target detection algorithms such as YOLO, the t values ​​in the image are obtained based on the aircraft's geometric dimensions. k Time of flight geographic coordinates W k Corresponding pixel coordinates k = 1, 2, 3, 4, ..., n;

[0031] Finally, based on the aircraft's geographical coordinates W k (x k ,y k ,z kThe camera's position coordinates (x0, y0, z0), the angles φ and θ between the camera's optical axis and the vertical direction, the focal length f, and the physical lengths dx and dy of the pixels on the camera's sensor are used to calculate t using a pinhole imaging model. k The ideal coordinates V of the aircraft in the pixel coordinate system k (u k ,v k ), k=1, 2, 3, 4,...,n;

[0032] Specifically, the steps for calculating the ideal coordinates of the aircraft in the pixel coordinate system in step 2) are as follows:

[0033] First, based on the angle φ between the camera's optical axis and the vertical direction, the angle θ between the camera's optical axis and the horizontal reference direction, and the aircraft's geographic coordinates W... k (x k ,y k ,z k The camera position coordinates (x0, y0, z0) are used to convert the aircraft's geographic coordinates into cylindrical coordinates (ρ) with the optical axis as the axis. k ,θ k ,l k ).

[0034] Then, using the pinhole imaging model, (ρ) k ,θ k ,l k Convert to polar coordinates on the imaging plane Then convert to ideal pixel coordinates (u k ,v k ),in Where f is the camera focal length, dx and dy are the physical lengths of the pixel on the camera sensor, and u0 and v0 are the offsets of the pixel coordinate system origin.

[0035] Where the aircraft's geographic coordinates W k (x k ,y k ,z k The camera position coordinates (x0, y0, z0) can be obtained directly from the positioning information, φ and θ can be measured directly by precision measuring instruments, and u0, v0, f, dx, and dy can be measured in the laboratory by optical methods.

[0036] 3) Classify the aircraft's coordinates at all levels, and then solve the correspondence matrix between the geographic coordinate level and the image pixel level by referring to the classified three-level aircraft coordinates. Discard coordinate points with obvious errors and use the average value as the final correspondence matrix.

[0037] First, the aircraft's geographic coordinates W k (x k ,yk ,z k ), k = 1, 2, 3, 4, ..., n; classified into m groups based on the classification method where linearly correlated coordinates belong to the same class;

[0038] Then, following the classification method described in S12, with time t k Using the pixel coordinates as a reference and ideal pixel coordinates V k (u k ,v k ), k = 1, 2, 3, 4, ..., n, corresponding to the m groups described in S31;

[0039] Finally, the m sets of ideal pixel coordinates V k Aircraft geographic coordinates W k Substitute W k =HV k In the process, the corresponding matrix H is solved. The m values ​​obtained are discarded if they have obvious errors, and the average of the remaining values ​​is used as the final corresponding matrix H.

[0040] 4) Find typical target points as reference points;

[0041] First, a suitable area around the aircraft's trajectory is manually selected so that the area contains clearly visible indicator lines.

[0042] Then, the edge features of the selected area are extracted using the Canny operator edge detection method to obtain edge pixels and their direction vectors. The Shi-Tomasi corner extraction algorithm is used to extract the pixel coordinates (u,v) of several indicator line corner points, based on the aircraft's relative coordinates V. k And the actual coordinates (x, y) of several corner points were calculated from the configuration data of the indicator lines. w ,y w ,z w ).

[0043] 5) Substitute the ideal pixel coordinates and actual pixel coordinates of the aircraft into the distortion equation to obtain the distortion matrix. Then, based on the correspondence between the pixel coordinate system and the world coordinate system, solve for several sets of rotation and translation matrices to achieve camera calibration.

[0044] First, the ideal pixel coordinates V of the m groups of aircraft are respectively... k Pixel coordinates Substitute into the solution equation

[0045]

[0046] in, We obtain m sets of k1, k2, k3, p1, p2, discard the data with obvious errors, take the average of the remaining data as the distortion parameter, and calculate the distortion matrix D.

[0047] Then, according to

[0048]

[0049] Several sets of rotation matrices R and translation matrices T are obtained, and the average value is taken as the final R and T, where D is the distortion removal matrix and I is the intrinsic parameter matrix.

[0050] 6) Real-time positions of typical target points calculated using the corresponding matrix;

[0051] First, select several corner points in the image that have good detectability and small detection error, and use the Shi-Tomasi corner point extraction algorithm to extract the pixel coordinates (u) of several line corner points. n ,v n According to the ideal pixel coordinates V of the aircraft k Aircraft geographic coordinates W k And, based on the known geometric constraints, calculate the actual coordinates (x, y) of several corner points. n ,y n ,z n );

[0052] Then, the actual coordinates (x) n ,y n ,z n Substitute the values ​​into formula (2) and solve for the coordinates. With actual coordinates (x) n ,y n ,z n ) Subtract each square one by one and find the average S of the sum of squares. For error;

[0053] Finally, let the error in the first detection be... At appropriate intervals, repeat the test and obtain the error. like If the value exceeds a certain threshold, repeat steps S1 to S6 to recalibrate the camera and correct the accumulated errors generated during camera use.

[0054] 7) Use a target recognition algorithm to identify the pixel coordinates of non-cooperative targets in the image, and then use the corresponding matrix obtained under multi-mode information reference to solve for the actual coordinates of the non-cooperative targets, thereby achieving target tracking;

[0055] First, pre-training is performed. A YOLO network is trained using the ImageNet dataset. To obtain abstract discriminative information about the target, a deep convolutional neural network is used for target feature extraction.

[0056] Then, the pre-trained model is used to initialize the network parameters of the YOLO model, and the VOC dataset is used to train and fine-tune the model.

[0057] The trained YOLO model is used for aircraft detection. The aircraft outline is extracted within the detection box and the centroid is obtained. The pixel coordinates corresponding to the aircraft's geographic coordinates are calculated using the centroid and the aircraft's size.

[0058] Then, substitute W t =HV t Solve for the actual coordinates W of the non-cooperative objective. t (x t ,y t ,z t );

[0059] Finally, an error term is added to the actual coordinates to... As coordinate values ​​with errors.

[0060] Specifically, in step 7), the YOLO model is first pre-trained. The first 20 convolutional layers, 1 average pooling layer, and 1 fully connected layer of DarkNet53 are used to train the YOLO network on the ImageNet dataset containing 1000 categories.

[0061] Then, a pre-trained model was used to initialize the network parameters of the first 20 convolutional layers of the YOLO model, and the YOLO model was then trained and fine-tuned using the VOC dataset with 20 categories.

[0062] Finally, the trained YOLO model is used for aircraft detection. The aircraft's outline is extracted within the detection box, and its centroid is obtained. The pixel coordinates corresponding to the aircraft's geographic coordinates are calculated using the centroid and the aircraft's dimensions.

[0063] Example

[0064] The specific implementation process of this project is as follows: Figure 1 As shown.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A camera self-calibration and target tracking method based on multi-modal information reference in a large-scale airport scene, characterized in that, Using the GBAS system, the geographical location information of an aircraft traveling along a designated line on the ground under large-scale monitoring is sampled at appropriate time intervals to obtain the aircraft's geographical coordinates at the corresponding moments. After determining the relative positional relationship between the geographical information and the aircraft, a target detection algorithm is used to obtain the pixel coordinates of the aircraft at the corresponding moments in the image, based on the aircraft's geometric dimensions. The system then uses the aircraft's actual coordinates and various camera parameters, including camera position coordinates, the angle between the camera's optical axis and the vertical direction, the angle with the horizontal reference direction, focal length, and the physical length of the pixel on the camera's sensor. A pinhole camera model is used to calculate the ideal coordinates of the aircraft in the pixel coordinate system at each moment. The aircraft's coordinates are then classified into geographical coordinates, pixel coordinates, and ideal coordinates. Finally, the correspondence matrix between the geographical coordinates and the ideal coordinates is calculated using the classified coordinates. Coordinates with significant errors are discarded, and the average value is used as the final correspondence matrix. A suitable area around the aircraft trajectory is selected, and edge features are extracted from the selected area using edge detection methods. A corner point extraction algorithm is used to extract the pixel coordinates of several line corner points. The actual coordinates of these line corner points are then calculated based on the aircraft's geographic coordinates and the configuration data of the indicator lines, serving as the actual reference for calibration. The ideal aircraft coordinates and pixel coordinates are substituted into the distortion equation to obtain the distortion matrix. Several sets of rotation and translation matrices are then calculated based on the correspondence between the pixel coordinate system and the world coordinate system to achieve camera calibration. Several line corner points with good detectability and small detection errors are selected in the image, and the corner point extraction algorithm is used to extract their pixel coordinates. The actual coordinates of these line corner points are calculated based on the aircraft's geographic coordinates, ideal coordinates, and geometric constraint multi-mode information. The calculated coordinate values ​​are then solved based on the correspondence between the pixel coordinate system and the world coordinate system. The calculated values ​​are subtracted from the actual coordinates one by one, and the average of the sum of squares is obtained. The square root of this sum is taken as the error. After appropriate intervals, the calibration is performed again with reference to new multi-mode information to correct the effects of accumulated errors generated during camera use, thus calibrating the camera calibration results. Target recognition algorithms are used to identify targets and detect the pixel coordinates of non-cooperative targets in the image. Then, the actual coordinates of the non-cooperative targets are solved using the corresponding matrix obtained under multi-modal information reference, thus achieving target tracking.

Images