Image data-based aircraft target identification and position calculation method
By calibrating the photoelectric rotary camera and using the YOLO-v5 model to identify the aircraft, and combining the camera parameters for coordinate conversion, the problem of traditional technology being difficult to monitor aircraft without ADS-B equipment is solved, and the aircraft is accurately identified and positional resolution is achieved. Improved monitoring capabilities.
Patent Information
- Application Number
- CN202410912664.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional aircraft surveillance technology is difficult to effectively monitor aircraft, birds or other unknown obstacles that are not equipped with ADS-B equipment, and radar-based surveillance costs and complex operations. The existing aircraft position solution method based on image vision has timeouts. Offset and geolocation solving complex problems.
By calibrating the photoelectric rotary camera, obtaining internal and external parameters, identifying the aircraft using the YOLO-v5 model, obtaining two-dimensional pixel coordinates, and combining camera parameters to convert coordinates to realize the aircraft's three-dimensional world coordinate solution.
It realizes accurate identification and positional calculation of the aircraft, improves the aircraft monitoring capabilities, solves the problem that the aircraft cannot be accurately positioned by the optoelectronic turntable, and improves the positional calculation efficiency.
Smart Images

Figure CN118918571A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of aircraft position calculation, and in particular to an aircraft target recognition and position calculation method based on image data. Background Art
[0002] In today's society, with the gradual opening of low-altitude airspace, in order to ensure the safety of low-altitude aircraft, the implementation of low-altitude airspace aircraft monitoring has become an inevitable trend in the development of air traffic control. In aircraft monitoring, the accurate identification of aircraft and the accurate acquisition of their positions play a vital role in the aircraft monitoring capability, so the target recognition technology and position solution method of aircraft have also been highly valued by researchers. Traditional monitoring methods always reveal some disadvantages. For example, ADS-B monitoring technology can only identify and obtain the positions of aircraft or cooperative targets equipped with ADS-B equipment, and cannot effectively monitor non-cooperative targets such as aircraft, birds or other unknown obstacles that are not equipped with ADS-B. Most of the existing technologies monitor non-cooperative targets through radar, but the deployment of radar is expensive and the operation is complicated. Image visual information has strong situational awareness in medium and short distance scenes. Target recognition and position acquisition of aircraft based on image data can lock aircraft targets more intuitively than ADS-B technology, and can achieve accurate identification and position solution of aircraft at a relatively low cost.
[0003] At present, the image-based position solution of aircraft mainly includes relative visual positioning and absolute visual positioning. The essential difference between these two methods is that relative visual positioning is achieved by matching adjacent frames in the image data, and absolute visual positioning is achieved by extracting image data at a certain moment as a reference frame to match other frames. However, relative visual positioning is susceptible to timeout offset, and absolute visual positioning has the problem of complex geographic location solution process. Therefore, studying an efficient and simple method for aircraft position solution is of great significance to improving aircraft monitoring capabilities. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide an aircraft target recognition and position solution method based on image data in view of the deficiencies of the above-mentioned prior art, so as to realize aircraft target recognition and position solution.
[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for identifying aircraft targets based on image data, calibrating the camera of the optoelectronic turntable, obtaining the internal and external parameters of the optoelectronic turntable camera; identifying the aircraft target and obtaining the two-dimensional pixel coordinates of the aircraft; according to the two-dimensional pixel coordinates of the aircraft, combined with the internal and external parameters of the optoelectronic turntable camera, the two-dimensional pixel coordinates are converted into three-dimensional world coordinates through a coordinate conversion algorithm, thereby obtaining the position of the aircraft. It includes the following steps:
[0006] Step 1: Calibrate the camera of the optoelectronic turntable through the camera calibration model to obtain the internal and external parameters of the optoelectronic turntable camera;
[0007] The specific method of calibrating the camera of the photoelectric turntable includes:
[0008] Step 1.1: Based on the pinhole imaging principle, construct the photoelectric turntable imaging model;
[0009] Step 1.2: Use the photoelectric turntable to collect chessboard images, and calibrate the photoelectric turntable camera using the chessboard calibration method;
[0010] Step 1.3: Using the calibration results of step 1.2, obtain the internal and external parameters of the optoelectronic turntable camera;
[0011] Step 2: Use the optoelectronic turntable to collect aircraft images, annotate anchor frames, and construct an aircraft dataset;
[0012] Step 3: Based on the aircraft data set in step 2, build a target detection model for pre-training to obtain a pre-trained target detection model;
[0013] Construct the Yolo-v5 model as the target detection model and perform aircraft recognition. First, initialize the weight of the Yolo-v5 model, use the aircraft as the training recognition target, set different model parameters, perform multiple rounds of training, and finally obtain the optimal model weight.
[0014] Step 4: Test the pre-trained object detection model and obtain the 2D pixel coordinates of the aircraft;
[0015] After obtaining the optimal model weight, the collected aircraft video is preprocessed, and the data is input into the trained target detection model to identify the aircraft target. After the aircraft is identified, the two-dimensional pixel coordinates of the aircraft are output;
[0016] Step 5: Based on the two-dimensional pixel coordinates of the aircraft, combined with the internal and external parameters of the optoelectronic turntable camera obtained in step 1, the three-dimensional world coordinates of the aircraft are calculated.
[0017] According to the two-dimensional pixel coordinates of the aircraft obtained in step 4, combined with the internal and external parameters of the optoelectronic turntable camera obtained in step 1, the two-dimensional pixel coordinates are converted into three-dimensional world coordinates through the coordinate conversion algorithm to obtain the position of the aircraft. The specific method is as follows:
[0018] Set the 3D world coordinates of the world coordinate system WCS to (x w ,y w ,z w ) T , the homogeneous coordinates of the camera coordinate system CCS are (x c ,y c ,z c ) T , the two-dimensional pixel coordinates are (u,v) T Since pixel coordinates are two-dimensional coordinates and world coordinates are three-dimensional coordinates, the two-dimensional pixel coordinates need to be increased by a known dimension during the conversion process, that is, a z c Value, z c Indicates depth, that is, the distance between the optoelectronic turntable and the aircraft;
[0019] (1) Convert from pixel coordinates to image coordinates;
[0020] First, perform a reverse translation, that is, move the origin of the pixel coordinate system to the origin of the image coordinate system, and then perform a reverse discretization, using the physical size of the pixel to convert the integer value coordinates in the pixel coordinate system to the floating point coordinates in the image coordinate system; the specific conversion process from pixel coordinates to image coordinates is shown in the following formula:
[0021]
[0022] Among them, (x, y) represents the image coordinates, (u, v) represents the pixel coordinates, and d x and d y Respectively represent the length of a single pixel in the x-direction and the y-direction, (u0, v0) represents the pixel coordinates of the principal point;
[0023] (2) Convert image coordinates to camera coordinates, including normalization, depth information acquisition, and back-projection;
[0024] First, the image coordinates are normalized to unit coordinates based on the internal parameters of the photoelectric turntable. Then, the depth information corresponding to a point on the image is obtained. Finally, the normalized coordinates and depth information are used for back-projection calculation to obtain the camera coordinates. The specific conversion process is shown in the following formula:
[0025]
[0026] (3) Convert camera coordinates to world coordinates;
[0027] The camera coordinates are transformed into the world coordinate system using the camera extrinsic matrix, including the camera's rotation matrix and translation matrix. The specific conversion process is shown in the following formula:
[0028]
[0029] Among them, R represents the rotation matrix of the camera, and T represents the translation matrix of the camera.
[0030] The beneficial effect of adopting the above technical solution is that: the present invention provides an aircraft target recognition and position solution method based on image data, which identifies the aircraft target through the YOLO-v5 model and outputs the two-dimensional pixel coordinates of the aircraft, and then converts the two-dimensional pixel coordinates of the aircraft target into three-dimensional world coordinates through coordinate conversion, thereby realizing accurate recognition and position solution of the aircraft, solving the problem that the photoelectric turntable cannot accurately locate the identified aircraft, and improving the efficiency of aircraft position solution. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A flow chart of a method for aircraft target recognition and position calculation based on image data provided by an embodiment of the present invention;
[0032] Figure 2 Schematic diagram of the imaging model of the photoelectric turntable provided in an embodiment of the present invention.
[0033] Figure 3 This is a diagram of the aircraft position solution results provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0035] In this embodiment, a method for aircraft target recognition and position calculation based on image data is provided. First, the camera of the optoelectronic turntable is calibrated to obtain the internal and external parameters of the optoelectronic turntable camera. Then, the aircraft target is recognized through the target detection model. After successful recognition, the two-dimensional pixel coordinates of the aircraft are output. Combined with the internal and external parameters of the optoelectronic turntable, the two-dimensional pixel coordinates of the aircraft are converted into three-dimensional world coordinates through a coordinate conversion algorithm. Figure 1 As shown, the specific steps include:
[0036] Step 1: Calibrate the camera of the optoelectronic turntable through the camera calibration model to obtain the internal and external parameters of the optoelectronic turntable camera;
[0037] In this embodiment, the calibration performed on the photoelectric turntable is mainly to calibrate the camera of the photoelectric turntable, and the specific method includes:
[0038] Step 1.1: Based on the pinhole imaging principle, construct the photoelectric turntable imaging model;
[0039] After the optoelectronic turntable recognizes the aircraft, the two-dimensional pixel coordinates of the aircraft need to be converted into three-dimensional world coordinates. However, during the conversion process, the optoelectronic turntable imaging model needs to be constructed to determine the projection relationship of the aircraft in the three-dimensional space in the two-dimensional image. The optoelectronic turntable imaging model is as follows: Figure 2 As shown. The internal parameters of the optoelectronic turntable camera reflect the relationship between the camera coordinate system (CCS) and the image coordinate system (ICS), and represent the optical and geometric characteristics of the camera, such as the principal point (image center), focal length, radial lens distortion, off-axis lens distortion, etc. The mathematical model of the camera's internal parameters can be expressed by a 3×3 matrix A, as shown in the following formula:
[0040]
[0041] in, and are the scale factors of the horizontal axis x and vertical axis y of the camera image plane, respectively. f represents the focal length of the camera, d x and d y is the physical distance in the horizontal and vertical directions on the image plane. x and c y are the horizontal and vertical offsets between the camera center and the optical axis, respectively. s is the tilt factor, which is zero when the camera image plane coordinate axes are not orthogonal to each other. The camera internal parameter calibration mainly solves four parameters (f x ,f y ,c x ,c y ). However, in the actual process, in order to make the result more accurate, the distortion coefficient D of the photoelectric turntable camera needs to be considered, as shown in the following formula:
[0042] D=(k1,k2,k3,p1,p2)
[0043] Among them, k1, k2, k3 represent the radial distortion coefficients of the optoelectronic turntable camera, and p1, p2 represent the tangential distortion coefficients of the optoelectronic turntable camera.
[0044] The external parameters represent the position and orientation of the camera in the world coordinate system (WCS), that is, the relationship between the unified camera coordinate system (CCS) and the world coordinate system (WCS). The external parameters include the rotation matrix R and the translation matrix T, as shown in the following formula:
[0045]
[0046] T=[t x t y tz ] T
[0047] Among them, r 11 、r 12 、r 13 They represent the projection of the X axis in the camera coordinate system onto the X, Y, and Z axes in the world coordinate system, respectively. 21 、r 22 、r 23 They represent the projection of the Y axis of the camera coordinate system on the X, Y, and Z axes in the world coordinate system, respectively. 31 、r 32 、r 33 Respectively represent the projection of the Z axis of the camera coordinate system on the X, Y, and Z axes in the world coordinate system. x ,t y ,t z They respectively represent the translation of the origin of the camera coordinate system on the X, Y, and Z axes of the world coordinate system.
[0048] By solving the external parameters, the camera coordinate system (CCS) and the world coordinate system (WCS) can be unified. Then, combined with the camera internal parameters, the coordinate systems can be unified, thereby realizing the mapping from 3D space to 2D image and the conversion from camera coordinates to world coordinates.
[0049] Step 1.2: Use the photoelectric turntable to collect chessboard images, and calibrate the photoelectric turntable camera using the chessboard calibration method;
[0050] After collecting chessboard images using the photoelectric turntable and obtaining images of the chessboard calibration plate at different angles, the chessboard images collected by the photoelectric turntable are detected using the OpenCV library in Python, and the internal and external parameters of the photoelectric turntable are calculated. Step 1.3: Using the calibration results of step 1.2, the internal and external parameters of the photoelectric turntable camera are obtained.
[0051] In this embodiment, the internal parameters of the photoelectric turntable camera are shown in Table 1, and the external parameters are shown in Table 2.
[0052] Table 1 Internal parameters of the photoelectric turntable
[0053]
[0054] Table 2 External parameters of the photoelectric turntable at different angles
[0055]
[0056]
[0057] The internal parameter matrix A of the photoelectric turntable camera is:
[0058]
[0059] The distortion coefficient D is:
[0060] D=(k1,k2,k3,p1,p2)
[0061] =(4.70073962e-01, -2.63346443e+01, -9.07409145e-03, -3.09539472e-04, 2.13681325e-01)
[0062] According to the rotation vector and translation vector of the optoelectronic turntable camera at different angles in Table 2, the average value is taken to finally obtain the rotation matrix R and translation vector T of the optoelectronic turntable camera:
[0063]
[0064] T=[t x t y t z ]
[0065] =[-0.10267773-0.085499221.35696672]
[0066] Step 2: Use the calibrated optoelectronic turntable to collect aircraft images, annotate anchor frames, and construct an aircraft dataset;
[0067] In the process of aircraft recognition, high-precision recognition requires a suitable anchor box, which is a set of initial regions that share a fixed size and aspect ratio. The more accurate the dataset anchor box is, the easier the model training is, and the closer the predicted bounding box is to the actual bounding box. Therefore, before training, the collected images need to be annotated to obtain the aircraft dataset;
[0068] Step 3: Based on the aircraft data set obtained in step 2, a target detection model is constructed for pre-training to obtain a pre-trained target detection model;
[0069] The Yolo model is an advanced real-time target detection model. The present invention takes into account that the Yolo-v5 model has fast detection speed, is easy to adjust, and is simple to deploy. Therefore, the Yolo-v5 model is constructed as a target detection model for aircraft recognition; first, the weight of the Yolo-v5 model is initialized, and the aircraft is used as a training recognition target. According to the data set and actual requirements, the aircraft is successfully identified through the Yolo-v5 model, and different model parameters are set at the same time for multiple rounds of training to finally obtain the optimal model weight.
[0070] Step 4: Test the pre-trained object detection model and obtain the 2D pixel coordinates of the aircraft;
[0071] After obtaining the optimal model weights, the collected aircraft video is preprocessed, that is, the video is processed into separate frames and input into the trained target detection model frame by frame to identify the aircraft target. After the aircraft is identified, the two-dimensional pixel coordinates of the aircraft are output.
[0072] Step 5: Based on the two-dimensional pixel coordinates of the aircraft, combined with the internal and external parameters of the optoelectronic turntable camera obtained in step 1, the three-dimensional world coordinates of the aircraft are calculated.
[0073] According to the two-dimensional pixel coordinates of the aircraft obtained in step 4, combined with the internal and external parameters of the optoelectronic turntable obtained in step 1, the two-dimensional pixel coordinates of the aircraft are converted into three-dimensional world coordinates through a coordinate conversion algorithm to obtain the position of the aircraft. The specific method is as follows:
[0074] Assume that the 3D world coordinates of the aircraft in the world coordinate system WCS are (x w ,y w ,z w ) T , the homogeneous coordinates of the camera coordinate system CCS are (x c ,y c ,z c ) T , the two-dimensional pixel coordinates are (u,v) T Since pixel coordinates are two-dimensional coordinates and world coordinates are three-dimensional coordinates, the two-dimensional pixel coordinates need to be increased by a known dimension during the conversion process, usually by a z c Value, this value represents the depth depth, that is, the distance between the optoelectronic turntable and the aircraft.
[0075] (1) Convert from pixel coordinates to image coordinates;
[0076] The first is the reverse translation, that is, moving the origin of the pixel coordinate system (usually the pixel in the upper left corner) to the origin of the image coordinate system (usually the center of the imaging plane), and then the reverse discretization, converting the integer value coordinates in the pixel coordinate system to floating point coordinates in the image coordinate system, which needs to be done using the physical size of the pixel. The specific conversion process from pixel coordinates to image coordinates is shown in the following formula:
[0077]
[0078] Among them, (x, y) represents the image coordinates, (u, v) represents the pixel coordinates, and (u0, v0) represents the pixel coordinates of the camera principal point.
[0079] (2) Convert image coordinates to camera coordinates, including normalization, depth information acquisition, and back-projection.
[0080] First, the image coordinates are normalized to unit coordinates based on the internal parameters of the optoelectronic turntable, including the focal length and principal point of the camera. The depth information corresponding to a point on the image is obtained, and the normalized coordinates and depth information are used for back-projection calculation to obtain the camera coordinates. The specific conversion process is shown in the following formula.
[0081]
[0082] (3) Convert camera coordinates to world coordinates;
[0083] Use the external parameter matrix, including the rotation matrix and translation matrix of the camera, to transform the camera coordinates into the world coordinate system. The specific conversion process is shown in the following formula:
[0084]
[0085] Among them, R represents the rotation matrix of the camera, and T represents the translation matrix of the camera.
[0086] In this embodiment, according to the internal and external parameters of the photoelectric turntable, the two-dimensional pixel coordinates of the aircraft are converted into three-dimensional world coordinates. Figure 3 shown.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.
Claims
1. A method for aircraft target recognition based on image data, characterized in that: The camera of the optoelectronic turntable is calibrated to obtain the internal and external parameters of the optoelectronic turntable camera; the aircraft target is identified and the two-dimensional pixel coordinates of the aircraft are obtained; according to the two-dimensional pixel coordinates of the aircraft, combined with the internal and external parameters of the optoelectronic turntable camera, the two-dimensional pixel coordinates are converted into three-dimensional world coordinates through a coordinate conversion algorithm to obtain the position of the aircraft.
2. The method for aircraft target recognition based on image data according to claim 1, characterized in that: The following steps are involved: Step 1: Calibrate the camera of the optoelectronic turntable through the camera calibration model to obtain the internal and external parameters of the optoelectronic turntable camera; Step 2: Use the optoelectronic turntable to collect aircraft images, annotate anchor frames, and construct an aircraft dataset; Step 3: Based on the aircraft data set in step 2, build a target detection model for pre-training to obtain a pre-trained target detection model; Step 4: Test the pre-trained object detection model and obtain the 2D pixel coordinates of the aircraft; Step 5: Based on the two-dimensional pixel coordinates of the aircraft, combined with the internal and external parameters of the optoelectronic turntable camera obtained in step 1, the three-dimensional world coordinates of the aircraft are calculated.
3. The method for aircraft target recognition based on image data according to claim 2, characterized in that: The specific method of step 1 is: Step 1.1: Based on the pinhole imaging principle, construct the photoelectric turntable imaging model; Step 1.2: Use the photoelectric turntable to collect chessboard images, and calibrate the photoelectric turntable camera using the chessboard calibration method; Step 1.3: Using the calibration results of step 1.2, obtain the internal and external parameters of the optoelectronic turntable camera.
4. The method for aircraft target recognition based on image data according to claim 2, characterized in that: The step 3 constructs the Yolo-v5 model as the target detection model and performs aircraft recognition; first, the weight of the Yolo-v5 model is initialized, the aircraft is used as the training recognition target, and different model parameters are set to perform multiple rounds of training, and finally the optimal model weight is obtained.
5. The method for aircraft target recognition based on image data according to claim 4, characterized in that: After the optimal model weight is obtained in step 4, the collected aircraft video is preprocessed, and the data is input into the trained target detection model to identify the aircraft target. After the aircraft is identified, the two-dimensional pixel coordinates of the aircraft are output.
6. The method for aircraft target recognition based on image data according to claim 2, characterized in that: The step 5 converts the two-dimensional pixel coordinates of the aircraft obtained in step 4 into three-dimensional world coordinates through a coordinate conversion algorithm in combination with the internal and external parameters of the photoelectric turntable camera obtained in step 1, thereby obtaining the position of the aircraft; the specific method is as follows: Set the 3D world coordinates of the world coordinate system WCS to (x w ,y w ,z w ) T , the homogeneous coordinates of the camera coordinate system CCS are (x c ,y c ,z c ) T , the two-dimensional pixel coordinates are (u,v) T Since pixel coordinates are two-dimensional coordinates and world coordinates are three-dimensional coordinates, the two-dimensional pixel coordinates need to be increased by a known dimension during the conversion process, that is, a z c Value, z c Indicates depth, that is, the distance between the optoelectronic turntable and the aircraft; (1) Convert from pixel coordinates to image coordinates; First, perform a reverse translation, that is, move the origin of the pixel coordinate system to the origin of the image coordinate system, and then perform a reverse discretization, using the physical size of the pixel to convert the integer value coordinates in the pixel coordinate system to the floating point coordinates in the image coordinate system; the specific conversion process from pixel coordinates to image coordinates is shown in the following formula: Among them, (x, y) represents the image coordinates, (u, v) represents the pixel coordinates, dx and d y Respectively represent the length of a single pixel in the x-direction and the y-direction, (u0, v0) represents the pixel coordinates of the principal point; (2) Convert image coordinates to camera coordinates, including normalization, depth information acquisition, and back-projection; First, the image coordinates are normalized to unit coordinates based on the internal parameters of the photoelectric turntable. Then, the depth information corresponding to a point on the image is obtained. Finally, the normalized coordinates and depth information are used for back-projection calculation to obtain the camera coordinates. The specific conversion process is shown in the following formula: (3) Convert camera coordinates to world coordinates; The camera coordinates are transformed into the world coordinate system using the camera extrinsic matrix, including the camera's rotation matrix and translation matrix. The specific conversion process is shown in the following formula: Among them, R represents the rotation matrix of the camera, and T represents the translation matrix of the camera.
Citation Information
Cited By
Aircraft positioning method and device, aircraft, medium and program product
CN121475158A
