Method and system for measuring flight pose of aircraft

Abstract

The invention relates to a method and a system for measuring the flight pose of an aircraft. According to the technical scheme, high-precision measurement of the aircraft pose is achieved through multi-view camera array image collection, multi-camera imaging parameter calibration, image feature recognition, multi-view three-dimensional reconstruction, pose calculation and coordinate system conversion.

Description

Technical Field

[0001] The present invention relates to the field of pose measurement, and particularly to a method and a system for measuring the flight pose of an aircraft. Background Art

[0002] The accurate measurement of the flight pose of a large aircraft is crucial for evaluating flight safety and navigation accuracy. Traditional pose measurement methods, such as inertial navigation systems (INS) or global positioning systems (GPS), often cannot provide sufficiently high accuracy and reliability in the presence of dynamic high-frequency attitude changes and external environmental disturbances. In recent years, with the rapid development of computer vision and sensor technologies, spatial pose measurement technologies based on vision methods have received extensive attention due to their high accuracy, fast response, and strong anti-interference capabilities.

[0003] Currently, the prior art generally measures the aircraft pose based on binocular stereo vision technology. This measurement method uses the parallax of the binocular system to achieve aircraft pose measurement. Specifically, two cameras observe a spatial point from different perspectives, forming image points in the left and right camera images. According to the parallax and using the principle of triangulation, the three-dimensional coordinates of this spatial point can be located, thereby completing the three-dimensional coordinate reconstruction of multiple points on the aircraft. Subsequently, the spatial pose of the aircraft can be solved by fitting the three-dimensional point cloud.

[0004] However, most of the existing vision-based aircraft pose measurement methods use two camera perspectives. To ensure the pose measurement accuracy, they can only observe the flight pose in a small airspace at a short distance, and it is difficult to be applicable to the high-precision measurement of the aircraft pose in a long-distance and large-scale flight airspace.

[0005] Therefore, there is an urgent need for a flight pose measurement method and system that further improves the prior art. Summary of the Invention

[0006] The Summary of the Invention is provided to introduce in a simplified form some concepts that will be further described in the following Detailed Description section. The Summary of the Invention is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to assist in determining the scope of the claimed subject matter.

[0007] In view of the problems in the prior art, that is, to solve the problem that the existing vision-based measurement methods cannot meet the high-precision pose measurement requirements in large-range flight conditions such as the takeoff, landing, and low-altitude flight of large aircraft, the present invention proposes a measurement technology using a multi-view camera array. This technology can layout the camera array according to the flight track range of the aircraft, and use multi-view camera calibration, multi-view image feature recognition, feature point multi-view reconstruction, pose calculation, and coordinate system conversion to achieve accurate measurement of the flight track and attitude of the aircraft. Thus, the technical solution of the present invention can realize the acquisition of the flight process of the aircraft in a larger range, covering a distance of hundreds of meters, and can meet the pose measurement requirements in various conditions such as the low-altitude flight, takeoff, and landing of the aircraft.

[0008] Specifically, in one embodiment of the present invention, a system for measuring the flight pose of an aircraft is disclosed. The system may include:

[0009] A distributed camera array including multiple cameras, each of the multiple cameras being configured to capture multiple flight images of the unmanned aerial vehicle (UAV) and the aircraft within the field of view area of the camera;

[0010] A pixel coordinate acquisition device configured to determine the pixel coordinates of the feature points of the UAV and the aircraft in the multiple flight images through feature point detection, where the UAV itself is used as a feature point and the three-dimensional coordinates of the feature points on the aircraft in the aircraft coordinate system are known;

[0011] An imaging parameter calibration device configured to use a self-calibration algorithm to output a calibration result including the internal and external parameters of each camera based on the pixel coordinates of the feature points of the UAV;

[0012] A three-dimensional coordinate reconstruction device configured to reconstruct the three-dimensional rectangular coordinates of the feature points of the UAV and the aircraft in the camera coordinate system of each camera according to the calibration result and the pixel coordinates;

[0013] A GPS coordinate system conversion device configured to obtain the conversion relationship from the camera coordinate system to the GPS coordinate system according to the GPS three-dimensional coordinates of the feature points of the UAV in the GPS coordinate system during flight and the three-dimensional rectangular coordinates; and

[0014] An aircraft pose determination device configured to convert the three-dimensional rectangular coordinates of the feature points of the aircraft into GPS three-dimensional coordinates according to the conversion relationship, and obtain the conversion relationship between the aircraft coordinate system and the GPS coordinate system based on the GPS three-dimensional coordinates of the feature points of the aircraft and the three-dimensional coordinates in the aircraft coordinate system to determine the aircraft pose.

[0015] In one embodiment of the present invention, the distributed camera array can be divided into multiple groups of cameras and has a synchronous triggering setting, where each group of cameras images one or more identical field-of-view regions and the one or more identical field-of-view regions are not less than 50% of the total field-of-view region of the distributed camera array.

[0016] In one embodiment of the present invention, the drone can have a GPS positioning function and can be configured to fly along a predetermined trajectory within the total field-of-view region of the distributed camera array, and the number of positions during flight is not less than 2000.

[0017] In one embodiment of the present invention, the pixel coordinate acquisition device can be further configured to identify the pixel coordinates from the multiple flight images by using the gray centroid method.

[0018] In one embodiment of the present invention, the self-calibration algorithm can be configured to continuously adjust the pixel coordinates of the feature points of the drone and the internal parameters, distortion coefficients, and external parameters of each camera so that the reprojection error of the feature points in each camera is minimized, where the self-calibration algorithm can be the colmap open-source program.

[0019] In one embodiment of the present invention, the number of feature points on the aircraft is not less than 4 and can be arranged at positions that do not deform during flight, and the feature point recognition algorithm for identifying the pixel coordinates of each feature point of the aircraft can be selected based on the type of the feature point, and the type of the feature point can include corner points, coded points, white dots on a black background, and black dots on a white background.

[0020] In one embodiment of the present invention, the three-dimensional coordinate reconstruction device can be further configured to reconstruct the three-dimensional rectangular coordinates of the feature points of the drone and the aircraft in the camera coordinate system of each camera by using the principle of triangulation.

[0021] In one embodiment of the present invention, the GPS coordinate system conversion device can be further configured to use the rigid body transformation theory to obtain the conversion relationship from the camera coordinate system to the GPS coordinate system through the following operations:

[0022] Calculate the centroid of all the three-dimensional rectangular coordinates of the feature points of the drone;

[0023] Calculate the centroid of all the GPS three-dimensional coordinates of the feature points of the drone;

[0024] Decenter all the three-dimensional rectangular coordinates and all the GPS three-dimensional coordinates to solve the coordinate system transformation matrix;

[0025] Perform singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix;

[0026] Solve the translation vector based on the rotation matrix; and

[0027] Based on the translation vector and the rotation matrix, establish the conversion relationship from the camera coordinate system to the GPS coordinate system.

[0028] In an embodiment of the present invention, the aircraft pose determination device can be further configured to utilize the rigid body transformation theory to determine the aircraft pose through the following operations:

[0029] Calculate the centroid of all three-dimensional coordinates of the feature points of the aircraft in the aircraft coordinate system;

[0030] Calculate the centroid of all GPS three-dimensional coordinates of the feature points of the aircraft;

[0031] De-centroid all three-dimensional coordinates and all GPS three-dimensional coordinates to solve the coordinate system transformation matrix;

[0032] Perform singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix;

[0033] Solve the translation vector based on the rotation matrix; and

[0034] Based on the translation vector and the rotation matrix, determine the spatial pose of the aircraft coordinate system relative to the GPS coordinate system, where the rotation matrix represents the aircraft attitude and the translation vector represents the aircraft position.

[0035] In an embodiment of the present invention, wherein: the pixel coordinate acquisition device can be further configured to repeatedly determine the pixel coordinates of the feature points of the aircraft in the multiple flight images through feature point detection at each moment, the three-dimensional coordinate reconstruction device can be further configured to repeatedly reconstruct the three-dimensional rectangular coordinates of the feature points in the camera coordinate system according to the calibration result and the pixel coordinates of the feature points of the aircraft, and the aircraft pose determination device can be further configured to repeatedly convert the three-dimensional rectangular coordinates into GPS three-dimensional coordinates according to the conversion relationship at each moment and determine the aircraft pose based on the GPS three-dimensional coordinates of the feature points of the aircraft and the three-dimensional coordinates in the aircraft coordinate system.

[0036] In an embodiment of the present invention, the UAV can be monochromatic and the color can be distinguished from the background color of the total field of view area.

[0037] In an embodiment of the present invention, the feature points on the aircraft can be arranged on the fuselage and the wing roots.

[0038] In another embodiment of the present invention, a method for measuring the flight pose of an aircraft is disclosed, and the method includes:

[0039] Obtain multiple flight images of the drone within the field of view area of each camera through a distributed camera array, and determine the pixel coordinates of the feature points of the drone in these multiple flight images through feature point detection, where the drone itself is used as the feature point;

[0040] Use a self-calibration algorithm to output a calibration result including the internal and external parameters of each camera based on the pixel coordinates;

[0041] Reconstruct the three-dimensional rectangular coordinates of the feature points of the drone in the camera coordinate system of each camera according to the calibration result and the pixel coordinates, and obtain the conversion relationship from the camera coordinate system to the GPS coordinate system according to the three-dimensional rectangular coordinates and the GPS three-dimensional coordinates of the feature points of the drone in the GPS coordinate system during flight;

[0042] Obtain multiple flight images of the aircraft within the field of view area of each camera through the distributed camera array, and determine the pixel coordinates of the feature points of the aircraft in these multiple flight images through feature point detection, where the three-dimensional coordinates of the feature points on the aircraft in the aircraft coordinate system are known;

[0043] Reconstruct the three-dimensional rectangular coordinates of the feature points of the aircraft in the camera coordinate system of each camera according to the calibration result and the pixel coordinates of the feature points of the aircraft; and

[0044] Convert the three-dimensional rectangular coordinates of the feature points of the aircraft into GPS three-dimensional coordinates according to the conversion relationship, and obtain the conversion relationship between the aircraft coordinate system and the GPS coordinate system based on the GPS three-dimensional coordinates of the feature points of the aircraft and the three-dimensional coordinates in the aircraft coordinate system to determine the aircraft pose.

[0045] In an embodiment of the present invention, the pixel coordinates of the feature points of the drone and the aircraft can be identified from the multiple flight images using the gray centroid method.

[0046] In an embodiment of the present invention, the three-dimensional rectangular coordinates of the feature points of the drone and the aircraft in the camera coordinate system of each camera can be reconstructed using the principle of triangulation.

[0047] In an embodiment of the present invention, the conversion relationship from the camera coordinate system to the GPS coordinate system can be obtained by using the rigid body transformation theory through the following operations:

[0048] Calculate the centroid of all the three-dimensional rectangular coordinates of the feature points of the drone;

[0049] Calculate the centroid of all the GPS three-dimensional coordinates of the feature points of the drone;

[0050] Decenter all three-dimensional Cartesian coordinates and all GPS three-dimensional coordinates to solve for the coordinate system transformation matrix;

[0051] Perform singular value decomposition on the coordinate system transformation matrix to solve for the rotation matrix;

[0052] Solve for the translation vector based on the rotation matrix; and

[0053] Establish the transformation relationship from the camera coordinate system to the GPS coordinate system based on the translation vector and the rotation matrix.

[0054] In one embodiment of the present invention, the aircraft pose can be determined by the following operations using the rigid body transformation theory:

[0055] Calculate the centroid of all three-dimensional coordinates of the feature points of the aircraft in the aircraft coordinate system;

[0056] Calculate the centroid of all GPS three-dimensional coordinates of the feature points of the aircraft;

[0057] Decenter all three-dimensional coordinates and all GPS three-dimensional coordinates to solve for the coordinate system transformation matrix;

[0058] Perform singular value decomposition on the coordinate system transformation matrix to solve for the rotation matrix;

[0059] Solve for the translation vector based on the rotation matrix; and

[0060] Determine the spatial pose of the aircraft coordinate system relative to the GPS coordinate system based on the translation vector and the rotation matrix, where the rotation matrix represents the aircraft attitude and the translation vector represents the aircraft position.

[0061] After studying the following detailed description of the specific exemplary embodiments of the present invention in conjunction with the accompanying drawings, other aspects, features, and embodiments of the present invention will be apparent to those of ordinary skill in the art. Although the features of the present invention may be discussed below with respect to certain embodiments and drawings, all embodiments of the present invention may include one or more of the advantageous features discussed herein. In other words, although one or more embodiments may be discussed as having certain advantageous features, one or more of such features may also be used in accordance with the various embodiments of the present invention discussed herein. In a similar manner, although the exemplary embodiments may be discussed below as device, system, or method embodiments, it should be understood that such exemplary embodiments may be implemented in various devices, systems, and methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] To understand the manner in which the above-described features of the present disclosure are used in detail, the content briefly outlined above can be described more specifically with reference to various aspects, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings only illustrate certain typical aspects of the present disclosure and should not be considered as limiting its scope, as the description may allow other equally effective aspects.

[0063] Figure 1 is a schematic block diagram of a system for measuring the flight attitude of an aircraft according to an embodiment of the present disclosure.

[0064] Figure 2 is a schematic diagram of a multi-view camera array according to an embodiment of the present disclosure.

[0065] Figure 3 is a schematic diagram of a reference flight trajectory of an unmanned aerial vehicle according to an embodiment of the present disclosure.

[0066] Figure 4 is a schematic diagram of the arrangement of feature points on an aircraft according to an embodiment of the present disclosure.

[0067] Figure 5 is a schematic diagram of a reprojection error according to an embodiment of the present disclosure.

[0068] Figure 6 is a flowchart of a method for measuring the flight attitude of an aircraft according to an embodiment of the present disclosure. Detailed Embodiments

[0069] The following will describe each embodiment in more detail with reference to the accompanying drawings that form a part of the present invention and illustrate various specific exemplary embodiments. However, the embodiments can be implemented in many different forms and should not be construed as limiting the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be thorough and complete, and the scope of these embodiments will be fully conveyed to those of ordinary skill in the art. The embodiments can be implemented according to a method, system, or device. Therefore, these embodiments can adopt a hardware implementation form, a full software implementation form, or a form combining software and hardware aspects. Therefore, the following detailed embodiments are not restrictive.

[0070] The steps in each flowchart can be executed by hardware (e.g., a processor, an engine, a memory, a circuit), software (e.g., an operating system, an application, a driver, machine / processor executable instructions), or a combination thereof. As those of ordinary skill in the art will understand, the methods involved in the embodiments may include more or fewer steps than those shown.

[0071] In view of the problem that traditional flight pose measurement methods cannot meet the requirements of high-precision pose measurement under large-range flight conditions such as takeoff, landing, and low-altitude flight of large aircraft, the present invention proposes a measurement method and system using a multi-view camera array, which can layout the camera array according to the flight track range of the aircraft, and utilize multi-view camera calibration, multi-view image feature recognition, feature point multi-view reconstruction, pose calculation, and coordinate system conversion to achieve accurate measurement of the flight track and attitude of the aircraft.

[0072] In an exemplary embodiment of the present invention, when the flight altitude is less than or equal to 600 m, 9 cameras can be used to achieve pose measurement within a range of at least 500 m, the position measurement accuracy can reach 1 m, and the angle measurement accuracy can reach within 2°. If it is necessary to increase the measurement range and measurement altitude while ensuring the measurement accuracy, it can be achieved by increasing the number of cameras.

[0073] The following will describe various aspects of the present invention in detail.

[0074] Figure 1 It is a schematic block diagram of a system 100 for measuring the flight pose of an aircraft according to an embodiment of the present disclosure. The following description of the system 100 in Figure 1 will be more comprehensively illustrated in conjunction with Figures 2 - 5 to be more comprehensively illustrated.

[0075] As Figure 1 shown, the system 100 may include a distributed camera array 102, a pixel coordinate acquisition device 104, an imaging parameter calibration device 106, a three-dimensional coordinate reconstruction device 108, a GPS coordinate system conversion device 110, and an aircraft pose determination device 112.

[0076] In an embodiment of the present invention, the distributed camera array 102 may include multiple cameras, and each camera in the multiple cameras may be configured to capture multiple flight images of the unmanned aircraft and the aircraft within the field of view area of the camera. In the above embodiment of the present invention, the distributed camera array may be divided into multiple groups of cameras and have a synchronous trigger setting, where each group of cameras images one or more identical field of view areas and the one or more identical field of view areas are not less than 50% of the total field of view area of the distributed camera array.

[0077] Specifically, Figure 2Schematic diagram of a multi-view camera array according to an embodiment of the present disclosure. In the present invention, in order to accurately measure the flight track and attitude of an aircraft, when the flight altitude is less than or equal to 600 m, 9 cameras can be used to achieve pose measurement within a range of at least 500 m. The position measurement accuracy can reach 1 m, and the angle measurement accuracy can reach within 2°. If the measurement range and height need to be increased while ensuring the measurement accuracy, it can be achieved by increasing the number of cameras. The following takes 9 cameras as an example, but in other embodiments, any other appropriate number of cameras can be used according to the required accuracy.

[0078] As Figure 2 shown, combining the distributed camera array technology and the airspace requirements for aircraft flight attitude measurement, the layout of the multi-view distributed camera array can be completed. It is required that the overlapping area of the shooting airspace of a group of cameras (in the subsequent camera calibration technology, at least 3 cameras are required to image the same area. Therefore, Figure 2 #1, #2, and #3 in are a group, #4, #5, and #6 are a group, and #7, #8, and #9 are a group. However, in other embodiments, other appropriate numbers of cameras can also be used as a group according to the camera calibration technology adopted) is not less than 50% of the shooting field of view (in other embodiments, other appropriate shooting airspace overlapping percentages can be set according to the specific airspace requirements for flight attitude measurement). Synchronized image acquisition needs to be performed between the multi-view cameras.

[0079] In addition, the camera parameters (such as resolution, frame rate, etc.) in the multi-view distributed camera array need to be determined according to the aircraft flight speed, aircraft flight range, and pose measurement accuracy. According to the acquisition frame rate of the camera, the synchronization accuracy of the synchronization trigger device can be determined. If the camera acquisition frame rate is f, then the synchronization accuracy of the synchronization trigger device is not less than 0.1f. The placement angle between adjacent cameras should preferably be between 10° and 30°. The object distance of the camera placement and the distance between cameras need to be determined according to the shooting field of view and the angle between cameras. As can be understood by those skilled in the art, the placement angle between adjacent cameras is not limited to any specific angle or angle range, but any appropriate angle can be adopted as long as the purpose of obtaining suitable flight images is satisfied.

[0080] After the layout is completed, the UAV is controlled to fly within the entire field of view area of the distributed camera array. During the flight, each camera in the distributed camera array performs synchronized image acquisition of the UAV to obtain a series of image sequences including the UAV. For the image sequences obtained by each camera, it is required that the UAVs are evenly distributed in the field of view area, and the number of positions where the UAVs are located is not less than 2000.

[0081] Figure 3 Schematic diagram of a reference flight track of a UAV according to an embodiment of the present disclosure. In an embodiment of the present invention, as Figure 3As shown, the drone may have a GPS positioning function and may be configured to fly along a predetermined trajectory within the total field of view of the distributed camera array. In this embodiment, the GPS positioning accuracy of the drone needs to reach the ∼cm level, but other positioning accuracy levels may also be used in other embodiments. As can be understood by those skilled in the art, Figure 3 The flight trajectory shown is merely exemplary and non-limiting, and any other suitable predetermined flight trajectory may be adopted in other embodiments.

[0082] In one embodiment of the present invention, the drone may be monochromatic and the color may be distinguishable from the background color of the field of view. As can be appreciated by those skilled in the art, the color of the drone may not be limited to any specific color or number of colors, as long as it can be distinguished from the background color of the field of view.

[0083] Back to Figure 1 In one embodiment of the present invention, the pixel coordinate acquisition device 104 can be configured to determine the pixel coordinates of the feature points of the drone and the aircraft in the multiple flight images through feature point detection, wherein the drone itself is used as the feature point and the three-dimensional coordinates of the feature points on the aircraft in the aircraft coordinate system are known.

[0084] In this embodiment, the number of feature points on the aircraft is no less than 4 and can be arranged at positions that will not deform during flight (the deformation amount is lower than a threshold), and the feature point recognition algorithm used to identify the pixel coordinates of each feature point of the aircraft can be selected based on the type of the feature point, which may include corner points, coded points, white points on a black background, and black points on a white background.

[0085] Figure 4 FIG. 1 is a schematic diagram of the layout of feature points on an aircraft according to an embodiment of the present disclosure. Figure 4 As shown, four feature points can be arranged on the aircraft, and the locations of the feature points are distributed at places where the aircraft structure is rigid, such as two on the fuselage and two at the wing roots. As can be understood by those skilled in the art, the number of feature points on the aircraft is not limited to 4, and can also be less than or greater than 4, such as 3 or 5.

[0086] In the above embodiment, the pixel coordinate acquisition device 104 can be further configured to identify the pixel coordinates from the multiple flight images using the grayscale centroid method. Specifically, the pixel coordinates of the feature points in the image sequence taken by each camera can be extracted through the feature point detection algorithm. The file of pixel coordinates can be defined as "TXY", which means the pixel coordinates of the Xth feature point in the image taken by the Yth camera at time T. As can be understood by those skilled in the art, any other suitable method can be used to identify pixel coordinates, not limited to the grayscale centroid method.

[0087] Return to Figure 1 , in an embodiment of the present invention, the imaging parameter calibration device 106 may be configured to use a self-calibration algorithm to output a calibration result including the internal and external parameters of each camera based on the pixel coordinates of the feature points of the drone. In this embodiment, the self-calibration algorithm may be configured to continuously adjust the pixel coordinates of the feature points of the drone and the internal parameters, distortion coefficients, and external parameters of each camera so that the reprojection error of the feature points in each camera is minimized, where the self-calibration algorithm may be the colmap open-source program or any other suitable calibration program. In this embodiment, the pixel coordinates of the identified drone feature points may be written into an executable file and input into the self-calibration algorithm to output the internal and external parameters of each camera (the external parameters may be based on a reference camera in the camera array (any camera can be designated as the reference camera), or other bases may be used). Specifically, the self-calibration method refers to a technique for camera calibration that only uses the matching feature points in the images captured by the camera without the need for a marker with known three-dimensional coordinates. This method requires at least 3 cameras to image the same area, but other suitable methods may require different numbers of cameras to image the same area.

[0088] Specifically, Figure 5 is a schematic diagram of the reprojection error according to an embodiment of the present disclosure. As Figure 5 shown, the core of the self-calibration method is to continuously adjust the pixel coordinates (world coordinates) of the feature points and the internal parameters, distortion coefficients, and external parameters of each camera so that the reprojection error of the feature points in each camera (the error e between the pixel coordinates of the drone feature points in the flight image and the reprojection point of the feature points in the image plane) x is minimized. The reprojection error is as Figure 5 shown and can be expressed as the following formula:

[0089]

[0090] where X is the feature point and P is the projection matrix, representing the internal and external parameters of the camera.

[0091] Return to Figure 1 , in an embodiment of the present invention, the three-dimensional coordinate reconstruction device 108 may be configured to reconstruct the three-dimensional rectangular coordinates of the feature points of the drone and the aircraft in the camera coordinate system of each camera according to the calibration result and the pixel coordinates. In the above embodiment of the present invention, the three-dimensional coordinate reconstruction device 108 may be further configured to use the triangulation principle to reconstruct the three-dimensional rectangular coordinates of the feature points of the drone and the aircraft in the camera coordinate system of each camera. Specifically, the triangulation principle can be expressed as the following formula:

[0092]

[0093] It can be noted that when there are more than two cameras observing the same feature point (as shown in Figure 2 ), the 3D reconstruction result can be obtained by taking the average of multiple triangulations. Specifically, according to the feature point pixel coordinate file "T-X-Y", at a certain moment T, the cameras Y1, Y2,... that recognize the X feature point are identified. Accordingly, based on the imaging parameters and pixel coordinates of the cameras Y1, Y2,..., the 3D rectangular coordinates of the feature point in the camera coordinate system are reconstructed using the triangulation principle.

[0094] As can be understood by those skilled in the art, in other embodiments of the present invention, any other suitable method may also be used to obtain the 3D rectangular coordinates of the feature point in the camera coordinate system, not limited to the triangulation method.

[0095] In one embodiment of the present invention, the GPS coordinate system conversion device 110 may be configured to obtain the conversion relationship from the camera coordinate system to the GPS coordinate system based on the GPS 3D coordinates and 3D rectangular coordinates of the feature points of the unmanned aerial vehicle in the GPS coordinate system during flight. In this embodiment, the GPS coordinate system conversion device 110 may be further configured to use the rigid body transformation theory to obtain the conversion relationship from the camera coordinate system to the GPS coordinate system through the following operations: calculating the centroid of all 3D rectangular coordinates of the feature points of the unmanned aerial vehicle; calculating the centroid of all GPS 3D coordinates of the feature points of the unmanned aerial vehicle; de-centering all 3D rectangular coordinates and all GPS 3D coordinates to solve the coordinate system conversion matrix; performing singular value decomposition on the coordinate system conversion matrix to solve the rotation matrix; solving the translation vector based on the rotation matrix; and establishing the conversion relationship from the camera coordinate system to the GPS coordinate system based on the translation vector and the rotation matrix.

[0096] Specifically, assume that the 3D rectangular coordinates of the hovering position of the unmanned aerial vehicle in the camera coordinate system are x 1 , x 2 ,..., x n (n being not less than 4 is sufficient), and the corresponding GPS 3D coordinates in the GPS coordinate system are x 1 ￠, x￠ 2 ,..., x￠ n . First, calculate the centroid of all feature points in the camera coordinate system, as shown in the following formula:

[0097]

[0098] Similarly, calculate the centroid of all feature points in the GPS coordinate system, as shown in the following formula:

[0099]

[0100] Then, de - center the coordinates of all feature points as shown in the following formula:

[0101]

[0102] Subsequently, solve the coordinate system transformation matrix H as shown in the following formula:

[0103]

[0104] And perform singular value decomposition on H to solve the rotation matrix R as shown in the following formula:

[0105]

[0106] Then solve the translation vector T as shown in the following formula:

[0107] T = C′ - RC (8)

[0108] Finally, establish the conversion relationship between the x - direction of the camera coordinate system and the x′ - direction of the GPS coordinate system as shown in the following formula:

[0109] x′ = Rx+T (9)

[0110] As those skilled in the art can understand, in other embodiments of the present invention, any other suitable method may also be used to determine the conversion relationship from the camera coordinate system to the GPS coordinate system, and it is not limited to the above rigid - body transformation theory.

[0111] In an embodiment of the present invention, the aircraft pose determination device 112 may be configured to convert the three - dimensional rectangular coordinates of the feature points of the aircraft into GPS three - dimensional coordinates according to the above conversion relationship. Subsequently, the aircraft pose determination device 112 may further be configured to obtain the conversion relationship between the aircraft coordinate system and the GPS coordinate system based on the GPS three - dimensional coordinates of the feature points of the aircraft and the three - dimensional coordinates in the aircraft coordinate system to determine the aircraft pose.

[0112] In the above - mentioned embodiment of the present invention, the aircraft pose determination device 112 may be further configured to use the above rigid - body transformation theory to determine the aircraft pose through the following operations: calculate the centroid of all three - dimensional coordinates of the feature points of the aircraft in the aircraft coordinate system; calculate the centroid of all GPS three - dimensional coordinates of the feature points of the aircraft; de - center all three - dimensional coordinates and all GPS three - dimensional coordinates to solve the coordinate system transformation matrix; perform singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix; solve the translation vector based on the rotation matrix; and determine the spatial pose of the aircraft coordinate system relative to the GPS coordinate system based on the translation vector and the rotation matrix, where the rotation matrix can represent the aircraft attitude and can be used to solve the yaw angle, roll angle, and pitch angle of the aircraft, and the translation vector can represent the aircraft position.

[0113] In one embodiment of the present invention, the pixel coordinate acquisition device 104, the three-dimensional coordinate reconstruction device 108, and the aircraft pose determination device 112 may be further configured to repeatedly perform their respective corresponding operations at each moment, that is, cyclically perform the acquisition of the pixel coordinates of the aircraft feature points, the reconstruction of the three-dimensional rectangular coordinates of the aircraft feature points, and the determination of the aircraft pose, so as to obtain the aircraft pose at each moment.

[0114] Figure 6 It is a flowchart of a method 600 for measuring the flight pose of an aircraft according to an embodiment of the present disclosure.

[0115] As Figure 6 shown, the method 600 starts at step 602, and a plurality of flight images of the UAV within the field of view area of each camera are acquired through a distributed camera array including multiple cameras, and the pixel coordinates of the feature points of the UAV in the plurality of flight images are determined through feature point detection, where the UAV itself is used as the feature point. In one embodiment of the present invention, the pixel coordinates of the feature points of the UAV and the aircraft may be identified from the plurality of flight images by using the gray centroid method.

[0116] Next, the method 600 proceeds to step 604, and using a self-calibration algorithm, a calibration result including the internal parameters and external parameters of each camera is output based on the pixel coordinates.

[0117] Then, the method 600 proceeds to step 606, and based on the calibration result and the pixel coordinates, the three-dimensional rectangular coordinates of the feature points of the UAV in the camera coordinate system of each camera are reconstructed, and based on the three-dimensional rectangular coordinates and the GPS three-dimensional coordinates of the feature points of the UAV in the GPS coordinate system during flight, the conversion relationship from the camera coordinate system to the GPS coordinate system is obtained. In one embodiment of the present invention, the three-dimensional rectangular coordinates of the feature points of the UAV and the aircraft in the camera coordinate system of each camera may be reconstructed by using the principle of triangulation. In one embodiment of the present invention, the conversion relationship from the camera coordinate system to the GPS coordinate system may be obtained by using the rigid body transformation theory through the following operations: calculating the centroid of all the three-dimensional rectangular coordinates of the feature points of the UAV; calculating the centroid of all the GPS three-dimensional coordinates of the feature points of the UAV; de-centering all the three-dimensional rectangular coordinates and all the GPS three-dimensional coordinates to solve the coordinate system transformation matrix; performing singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix; solving the translation vector based on the rotation matrix; and establishing the conversion relationship from the camera coordinate system to the GPS coordinate system based on the translation vector and the rotation matrix.

[0118] Subsequently, method 600 proceeds to step 608, where multiple flight images of the aircraft within the field of view area of each camera are acquired by the distributed camera array, and the pixel coordinates of the feature points of the aircraft in the multiple flight images are determined through feature point detection, where the three-dimensional coordinates of the feature points on the aircraft in the aircraft coordinate system are known.

[0119] Next, method 600 proceeds to step 610, where the three-dimensional rectangular coordinates of the feature points of the aircraft in the camera coordinate system of each camera are reconstructed based on the calibration result and the pixel coordinates of the feature points of the aircraft.

[0120] Finally, method 600 proceeds to step 612, where the three-dimensional rectangular coordinates of the feature points of the aircraft are converted into GPS three-dimensional coordinates according to the conversion relationship, and the conversion relationship between the aircraft coordinate system and the GPS coordinate system is obtained based on the GPS three-dimensional coordinates of the feature points of the aircraft and the three-dimensional coordinates in the aircraft coordinate system to determine the aircraft pose. In an embodiment of the present invention, the aircraft pose can be determined by using the rigid body transformation theory through the following operations: calculating the centroid of all three-dimensional coordinates of the feature points of the aircraft in the aircraft coordinate system; calculating the centroid of all GPS three-dimensional coordinates of the feature points of the aircraft; de-centering all three-dimensional coordinates and all GPS three-dimensional coordinates to solve the coordinate system transformation matrix; performing singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix; solving the translation vector based on the rotation matrix; and determining the spatial pose of the aircraft coordinate system relative to the GPS coordinate system based on the translation vector and the rotation matrix, where the rotation matrix represents the aircraft attitude and the translation vector represents the aircraft position.

[0121] In an embodiment of the present invention, steps 608-612 can be cyclically executed at each moment to obtain the aircraft pose at each moment.

[0122] After step 612, method 600 ends.

[0123] In summary, the technical solution proposed by the present invention can achieve the acquisition of the flight process of the aircraft in a larger range by adopting the layout method of the distributed camera array, covering a distance of hundreds of meters, and can meet the pose measurement under various working conditions such as low-altitude flight, takeoff, and landing of the aircraft. In addition, by adopting the multi-camera self-calibration technology, the present invention can solve the multi-camera calibration in a large-range flight airspace, without relying on precise calibration objects, greatly reducing the calibration cost and improving the flexibility and accuracy of calibration.

[0124] The above references describe embodiments of the present invention in terms of block diagrams and / or operational descriptions of methods, systems, and computer program products according to embodiments of the present invention. The functions / actions noted in the blocks may occur in a different order than any flowchart shown. For example, depending on the functions / actions involved, two consecutive blocks shown may actually be executed substantially simultaneously, or these blocks may sometimes be executed in the reverse order.

[0125] As described above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A system for measuring the flight posture of an aircraft, the system comprising: a distributed camera array including a plurality of cameras, each camera of the plurality of cameras being configured to capture a plurality of flight images of drones and aircraft within a field of view area of ​​the camera; a pixel coordinate acquisition device configured to determine the pixel coordinates of the feature points of the drone and the aircraft in the plurality of flight images by feature point detection, wherein the drone itself is used as the feature point and the three-dimensional coordinates of the feature points on the aircraft in the aircraft coordinate system are known; An imaging parameter calibration device, configured to use a self-calibration algorithm to output a calibration result including an intrinsic parameter and an extrinsic parameter of each camera based on the pixel coordinates of the feature points of the drone; a three-dimensional coordinate reconstruction device, configured to reconstruct the three-dimensional rectangular coordinates of the feature points of the drone and the aircraft in the camera coordinate system of each camera according to the calibration results and the pixel coordinates; A GPS coordinate system conversion device, which is configured to obtain a conversion relationship from the camera coordinate system to the GPS coordinate system according to the GPS three-dimensional coordinates of the feature points of the drone in the GPS coordinate system during flight and the three-dimensional rectangular coordinates; as well as An aircraft attitude determination device is configured to convert the three-dimensional rectangular coordinates of the feature points of the aircraft into GPS three-dimensional coordinates according to the conversion relationship, and obtain the conversion relationship between the aircraft coordinate system and the GPS coordinate system based on the GPS three-dimensional coordinates of the feature points of the aircraft and the three-dimensional coordinates in the aircraft coordinate system to determine the aircraft attitude.

2. The system of claim 1, wherein the distributed camera array is divided into a plurality of groups of cameras and has a synchronized trigger setting, wherein each group of cameras images one or more identical field of view areas and the one or more identical field of view areas are not less than 50% of the total field of view area of ​​the distributed camera array.

3. The system of claim 1, wherein the drone has a GPS positioning function and is configured to fly along a predetermined trajectory within the total field of view of the distributed camera array, and the number of positions it takes during flight is no less than 2,000. 4 . The system of claim 1 , wherein the pixel coordinate acquisition device is further configured to identify the pixel coordinates from the plurality of flight images using a grayscale centroid method.

5. The system of claim 1, wherein the self-calibration algorithm is configured to continuously adjust the pixel coordinates of the feature points of the drone and the intrinsic parameters, distortion coefficients and extrinsic parameters of each camera so that the reprojection error of the feature points in each camera is minimized, wherein the self-calibration algorithm is a colmap open source program.

6. The system of claim 1 , wherein the number of feature points on the aircraft is no less than 4 and they are arranged at positions that will not deform during flight, and wherein a feature point recognition algorithm for identifying the pixel coordinates of each feature point of the aircraft is selected based on the type of the feature point, the type of feature point comprising corner points, coded points, white points on a black background, and black points on a white background.

7. The system of claim 1, wherein the three-dimensional coordinate reconstruction device is further configured to utilize a triangulation principle to reconstruct the three-dimensional rectangular coordinates of the feature points of the drone and the aircraft in the camera coordinate system of each camera.

8. The system of claim 1, wherein the GPS coordinate system conversion device is further configured to obtain the conversion relationship from the camera coordinate system to the GPS coordinate system by using rigid body transformation theory through the following operations: Calculating the centroid of all three-dimensional rectangular coordinates of the feature points of the drone; Calculate the centroid of all GPS three-dimensional coordinates of the feature points of the drone; De-centroidalize all 3D rectangular coordinates and all GPS 3D coordinates to solve the coordinate system transformation matrix; Performing singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix; Solving the translation vector based on the rotation matrix; as well as A transformation relationship from the camera coordinate system to the GPS coordinate system is established based on the translation vector and the rotation matrix.

9. The system of claim 1, wherein the aircraft posture determination device is further configured to determine the aircraft posture by using rigid body transformation theory through the following operations: Calculating the centroid of all three-dimensional coordinates of the characteristic points of the aircraft in the aircraft coordinate system; Calculate the centroid of all GPS three-dimensional coordinates of the characteristic points of the aircraft; De-centroidalize all 3D coordinates and all GPS 3D coordinates to solve the coordinate system transformation matrix; Performing singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix; Solving the translation vector based on the rotation matrix; as well as The spatial position of the aircraft coordinate system relative to the GPS coordinate system is determined based on the translation vector and the rotation matrix, wherein the rotation matrix represents the aircraft attitude and the translation vector represents the aircraft position.

10. A method for measuring a flight attitude of an aircraft, the method comprising: Acquiring a plurality of flight images of the drone within a field of view of each camera through a distributed camera array including a plurality of cameras and determining pixel coordinates of feature points of the drone in the plurality of flight images through feature point detection, wherein the drone itself is used as a feature point; Using a self-calibration algorithm, outputting a calibration result including an intrinsic parameter and an extrinsic parameter of each camera based on the pixel coordinates; Reconstructing the three-dimensional rectangular coordinates of the feature points of the drone in the camera coordinate system of each camera according to the calibration results and the pixel coordinates, and obtaining a conversion relationship from the camera coordinate system to the GPS coordinate system according to the three-dimensional rectangular coordinates and the GPS three-dimensional coordinates of the feature points of the drone in the GPS coordinate system during flight; Acquire multiple flight images of the aircraft in the field of view of each camera through the distributed camera array and determine the pixel coordinates of the feature points of the aircraft in the multiple flight images through feature point detection, wherein the three-dimensional coordinates of the feature points on the aircraft in the aircraft coordinate system are known; Reconstructing the three-dimensional rectangular coordinates of the feature points of the aircraft in the camera coordinate system of each camera according to the calibration results and the pixel coordinates of the feature points of the aircraft; as well as The three-dimensional rectangular coordinates of the feature points of the aircraft are converted into GPS three-dimensional coordinates according to the conversion relationship, and based on the GPS three-dimensional coordinates of the feature points of the aircraft and the three-dimensional coordinates in the aircraft coordinate system, the conversion relationship between the aircraft coordinate system and the GPS coordinate system is obtained to determine the aircraft posture.

11. The method of claim 10, wherein the pixel coordinates of the feature points of the drone and the aircraft are identified from the plurality of flight images using a grayscale centroid method.

12. The method of claim 10, wherein the three-dimensional rectangular coordinates of the feature points of the drone and the aircraft in the camera coordinate system of each camera are reconstructed using the triangulation principle.

13. The method of claim 10, wherein the transformation relationship from the camera coordinate system to the GPS coordinate system is obtained by using rigid body transformation theory through the following operations: Calculating the centroid of all three-dimensional rectangular coordinates of the feature points of the drone; Calculate the centroid of all GPS three-dimensional coordinates of the feature points of the drone; De-centroidalize all 3D rectangular coordinates and all GPS 3D coordinates to solve the coordinate system transformation matrix; Performing singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix; Solving the translation vector based on the rotation matrix; as well as A transformation relationship from the camera coordinate system to the GPS coordinate system is established based on the translation vector and the rotation matrix.

14. The method of claim 10, wherein the aircraft posture is determined by using rigid body transformation theory through the following operations: Calculating the centroid of all three-dimensional coordinates of the characteristic points of the aircraft in the aircraft coordinate system; Calculate the centroid of all GPS three-dimensional coordinates of the characteristic points of the aircraft; De-centroidalize all 3D coordinates and all GPS 3D coordinates to solve the coordinate system transformation matrix; Performing singular value decomposition on the coordinate system transformation matrix to solve the rotation matrix; Solving the translation vector based on the rotation matrix; as well as The spatial position of the aircraft coordinate system relative to the GPS coordinate system is determined based on the translation vector and the rotation matrix, wherein the rotation matrix represents the aircraft attitude and the translation vector represents the aircraft position.

Images