Method and system for locating airport surface targets

Abstract

The present invention discloses a method and system for locating targets at an airport scene. The method comprises: constructing a point cloud database; the point cloud database comprises template point clouds of different targets of various aircraft types at the airport scene; detecting visible light images of various aircraft types at the airport scene collected in real time through a trained landmark detection model to obtain landmark point information of targets of different categories; restoring the landmark point information to a landmark point cloud based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time; matching the template point cloud with the landmark point cloud to calculate a rotation and translation matrix; the rotation and translation matrix comprises a rotation matrix and a translation matrix; and calculating the coordinates of the target location point based on the rotation and translation matrix. The present invention combines target detection, landmark point detection and depth information in computer vision to achieve precise positioning of key points of large targets for the purpose of accurate perception of targets at the airport scene.

Description

Technical Field

[0001] The present invention relates to the field of airport safety technology, and in particular to a method and system for locating an airport scene target. Background Art

[0002] With the development of my country's civil aviation industry, the number of fleets at various airports is increasing rapidly. Most airports are in a state of saturation, ground traffic congestion is increasing, and the support capacity of airport service personnel is becoming increasingly insufficient. These reasons have led to the occurrence of airport surface scraping or dangerous approaching incidents in recent years, such as scraping between aircraft passenger ladders and aircraft doors, and between corridors and aircraft doors. In severe cases, aircraft may approach or collide with each other in a dangerous manner. At present, airport surface monitoring mainly relies on visual voice dispatching and surface monitoring radar, which can only judge the approximate position of the aircraft. When the distance is far, it is difficult to achieve effective early warning. The increasingly saturated airport traffic has increased the risk of the surface, which not only increases the workload of surface service personnel, but also affects the operation efficiency and benefits of the airport. In addition, the GPS information of the surface target can only provide the coordinate information of a certain point of the target (such as the GPS receiving antenna). The information is single, and it is difficult to obtain the information of the position of the key points of the target, such as the wing tip, the nose, etc., and it is difficult to judge the real position information between the targets. Therefore, unsafe accidents such as the wing tip scratching of the aircraft occur from time to time. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for locating an airport surface target, which are used to precisely locate the key points of the real spatial position of the target.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for locating an airport surface target, comprising:

[0006] Constructing a point cloud database; the point cloud database includes template point clouds of different targets of various aircraft types at the airport scene;

[0007] The trained landmark detection model is used to detect the visible light images of various aircraft types collected in real time at the airport scene to obtain the landmark information of different categories of targets.

[0008] Based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time, the marker point information is restored into a marker point cloud;

[0009] Matching the template point cloud with the marker point cloud, and calculating a rotation and translation matrix; the rotation and translation matrix includes a rotation matrix and a translation matrix;

[0010] The coordinates of the target object positioning point are calculated based on the rotation and translation matrix.

[0011] Optionally, the training process of the landmark detection model is as follows:

[0012] Collect visible light images of various aircraft types at the airport;

[0013] Annotating the target objects and the landmarks of the target objects in the visible light image to construct a training set;

[0014] The landmark point detection model is trained using the training set to obtain a trained landmark point detection model.

[0015] Optionally, based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time, restoring the marker point information of the different categories of targets into a marker point cloud specifically includes:

[0016] Align the depth image with the visible light image, take the RGB camera coordinate system as the reference coordinate system, and extract the depth value of the marker point;

[0017] The marker point information is restored into a marker point cloud based on the depth value and the camera projection model.

[0018] Optionally, matching the template point cloud with the marker point cloud to calculate a rotation and translation matrix specifically includes:

[0019] Calculate the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud;

[0020] Calculating a covariance matrix between the template point cloud and the marker point cloud based on the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud;

[0021] Based on the covariance matrix, the rotation and translation matrix is ​​calculated using the SVD decomposition method.

[0022] The present invention also provides an airport surface target positioning system, comprising:

[0023] A point cloud database construction module is used to construct a point cloud database; the point cloud database includes template point clouds of different targets of various aircraft types at the airport scene;

[0024] The landmark information detection module is used to detect the visible light images of various aircraft types collected in real time at the airport scene through the trained landmark detection model to obtain the landmark information of different categories of targets;

[0025] A marker point cloud recovery module is used to restore the marker point information into a marker point cloud based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time;

[0026] A rotation and translation matrix calculation module, used for matching the template point cloud with the marker point cloud and calculating a rotation and translation matrix; the rotation and translation matrix includes a rotation matrix and a translation matrix;

[0027] The target object positioning point coordinate calculation module is used to calculate the coordinates of the target object positioning point based on the rotation and translation matrix.

[0028] Optionally, it further includes: a training module, wherein the training module specifically includes:

[0029] Visible light image acquisition unit, used to collect visible light images of various aircraft types at the airport;

[0030] A training set construction unit, used to mark the target object and the landmark points of the target object in the visible light image to construct a training set;

[0031] The training unit is used to train the landmark detection model using the training set to obtain a trained landmark detection model.

[0032] Optionally, the landmark point cloud recovery module specifically includes:

[0033] A depth value extraction unit, used to align the depth image with the visible light image, and extract the depth value of the marker point using the RGB camera coordinate system as a reference coordinate system;

[0034] A restoration unit is used to restore the marker point information into a marker point cloud based on the depth value and the camera projection model.

[0035] Optionally, the rotation and translation matrix calculation module specifically includes:

[0036] A center point coordinate calculation unit, used to calculate the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud;

[0037] A covariance matrix calculation unit, used to calculate the covariance matrix between the template point cloud and the marker point cloud based on the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud;

[0038] The rotation and translation matrix calculation unit is used to calculate the rotation and translation matrix based on the covariance matrix by using the SVD decomposition method.

[0039] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0040] The present invention combines target detection and landmark point detection in computer vision with depth information to achieve precise positioning of key points of large-scale targets for the purpose of accurately perceiving targets on the airport scene. It can solve the problem that existing monitoring methods based on satellite positioning and scene surveillance radar positioning cannot locate key points of targets, improve the monitoring capability of active targets on the airport scene, and enhance the safety of airport scene operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A flow chart of the airport surface target positioning method provided by the present invention;

[0043] Figure 2 A schematic diagram of the airport surface target positioning method provided by the present invention;

[0044] Figure 3 This is a schematic diagram of the structure of the CenterNet model;

[0045] Figure 4 The template point clouds of some template aircraft models in the point cloud database, where (a) is the template point cloud of the A321 model, and (b) is the template point cloud of the B373 model;

[0046] Figure 5 This is a schematic diagram of the coordinates of the target object positioning point in the RGB camera coordinate system. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The purpose of the present invention is to provide a method and system for locating an airport surface target, which are used to precisely locate the key points of the real spatial position of the target.

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] like Figure 1-2 As shown, the airport scene target positioning method provided by the present invention comprises the following steps:

[0051] Step 101: construct a point cloud database; the point cloud database includes template point clouds of different targets of various aircraft types at the airport scene, such as Figure 4 (a) and Figure 4 (b) as shown.

[0052] The detailed template point cloud data of each aircraft model is scanned using a high-precision 3D scanner; for each type of aircraft, there are 5 landmark points used for detection, distributed in the left and right engine pod positions, the aircraft fuselage, and the connection between the rudder and the fuselage.

[0053] Step 102: Detect the visible light images of various aircraft types at the airport scene collected in real time through the trained landmark detection model to obtain landmark information of different categories of targets.

[0054] The training process of the landmark detection model includes:

[0055] (1) Collect visible light images of various aircraft types at the airport.

[0056] The data set is collected from the airport scene simulation platform. Since the data collected by the laser radar on the scene simulation platform is too sparse, the collection device used in this invention is an RGB-D camera. The device includes two cameras with different attributes, namely an RGB camera and a Depth camera. The RGB camera is used to collect visible light images of various aircraft models at the airport scene, and the Depth camera is used to collect the corresponding depth images.

[0057] (2) Labeling the target objects and their landmarks in the visible light image to construct a training set.

[0058] (3) Training the landmark point detection model using the training set to obtain a trained landmark point detection model.

[0059] The present invention uses a target detection model CenterNet with a simple structure and easy expansion. Figure 3 It is applied to scene target landmark detection, specifically:

[0060] 1) Data processing and heatmap generation: The input scale is the resized 512x512 size. The center point coordinates are calculated according to the target box, and the heatmap values ​​of the two-dimensional normal distribution are filled at the center point and the landmark points.

[0061]

[0062] Where x, y are the locations of the fill map, The coordinates of a center point or landmark point when downsampled to the feature map, is the target scale adaptive variance, and Y is the mapping value.

[0063] 2) Model training: The feature extraction network used by CenterNet is DLA34, which takes into account both the speed and accuracy of detection. The scale of the feature layer after extraction is 128x128. Then the features are regressed through six detection heads for heatmap center point, target box width and height, target center point offset, landmark point coordinates, etc. The main regression loss function is the improved focalloss. ;

[0064]

[0065] Where N is the number of samples, is the predicted value, Y xyc is the true value, α, β are the penalty parameters.

[0066] Use the trained model to detect the target object’s landmark points. Take the landmark point as an example, (x p ,y p ) is the coordinate information decoding output by the trained landmark detection model, To predict the coordinates of the landmark points, is the offset of the predicted coordinates.

[0067]

[0068] Step 103: Based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time, the marker point information is restored into a marker point cloud.

[0069] Since the data input during detection is a visible light image, and the subsequent point cloud recovery is also based on the visible light image, the RGB camera coordinate system is used as the reference coordinate system to align the points in the depth coordinate system to the RGB coordinate system. The specific pixel alignment process is as follows:

[0070] (1) Convert a point in the depth camera coordinate system to the world coordinate system

[0071] (2) World coordinate system conversion RGB camera coordinate system p cc =T w2c p w ;

[0072] (3) Conversion from RGB camera coordinate system to RGB image coordinate system

[0073] Among them, p dc 、p cc , are the coordinates of the depth camera coordinate system, the RGB camera coordinate system, and the RGB image coordinate system respectively; p w is the coordinate of a point in the world coordinate system; T w2d 、T w2c , K are the depth camera extrinsic matrix, RGB camera extrinsic matrix, and RGB camera intrinsic matrix respectively; z is the depth value in the z direction in the RGB camera coordinate system; the extracted corresponding pixel depth value information is the distance d from the target object landmark point to the optical center plane of the visible light camera.

[0074] According to the camera projection model, the marker point information is restored to the point cloud form in three-dimensional space for subsequent matching. The detected marker point information is two-dimensional information in the RGB image coordinate system, while the template point cloud in the point cloud database is three-dimensional information, so it is necessary to combine the RGB camera internal parameters to restore the detected marker point cloud to a three-dimensional point cloud form. The details are as follows:

[0075]

[0076] Where d is the depth value corresponding to the marker point, and x and y are the coordinates of the marker point in the image coordinate system. x =f·d x f y =f·d y , f is the focal length of the visible light camera, dx, dy are the length units of a pixel in the x and y directions. According to the RGB camera projection model, the spatial position x of the marker point in the RGB camera coordinate system can be obtained. c ,y c ,z c .

[0077] Step 104: Match the template point cloud with the marker point cloud, and calculate a rotation and translation matrix; the rotation and translation matrix includes a rotation matrix and a translation matrix.

[0078] First, calculate the centroid position of each point set, that is, the coordinates of the center point, and set the landmark point set to The template point set is Each point has a one-to-one correspondence, the number is N, and the center point is c A , c B :

[0079]

[0080] According to the position of the center point, the point set is re-centered and the covariance matrix H between the two point sets is calculated:

[0081]

[0082]

[0083] Get the rotation and translation matrix between two point sets based on singular value decomposition:

[0084] [U,S,V]=SVD(H)

[0085] R=VU T , t=-R·c A +c B

[0086] Among them, R is the rotation matrix and t is the translation matrix, which are used for matching between the template point cloud and the landmark point cloud.

[0087] Step 105: Calculate the coordinates of the target object's location point based on the rotation and translation matrix.

[0088] Select the position P(P) to be located from the template point cloud 1 ,...P m ), a total of m position points, which are used as the positioning points of the target. According to the rotation and translation matrix Rt, the points to be positioned in the template point cloud are transformed to the RGB camera coordinate system:

[0089] C xyz (c 1 ,c 2 ,...c m )=R·P+t

[0090] C xyz is the coordinates of the positioning point in the RGB camera coordinate system, that is, the position information of the positioning point in the RGB camera coordinate system. The coordinate results of the target object positioning point are as follows Figure 5 The positioning points can be selected according to different application backgrounds. According to the scene background applied by this method, the selected positioning points include the upper and lower wing tips, horizontal tail, rudder, etc. of the aircraft.

[0091] The present invention also provides an airport surface target positioning system, comprising:

[0092] A point cloud database construction module is used to construct a point cloud database; the point cloud database includes template point clouds of different targets of various aircraft types at the airport scene;

[0093] The landmark information detection module is used to detect the visible light images of various aircraft types collected in real time at the airport scene through the trained landmark detection model to obtain the landmark information of different categories of targets;

[0094] A marker point cloud recovery module is used to restore the marker point information into a marker point cloud based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time;

[0095] A rotation and translation matrix calculation module, used for matching the template point cloud with the marker point cloud and calculating a rotation and translation matrix; the rotation and translation matrix includes a rotation matrix and a translation matrix;

[0096] The target object positioning point coordinate calculation module is used to calculate the coordinates of the target object positioning point based on the rotation and translation matrix.

[0097] The above system also includes: a training module, which specifically includes:

[0098] Visible light image acquisition unit, used to collect visible light images of various aircraft types at the airport;

[0099] A training set construction unit, used to mark the target object and the landmark points of the target object in the visible light image to construct a training set;

[0100] The training unit is used to train the landmark detection model using the training set to obtain a trained landmark detection model.

[0101] The landmark point cloud recovery module specifically includes:

[0102] A depth value extraction unit, used to align the depth image with the visible light image, and extract the depth value of the marker point using the RGB camera coordinate system as a reference coordinate system;

[0103] A restoration unit is used to restore the marker point information into a marker point cloud based on the depth value and the camera projection model.

[0104] The rotation and translation matrix calculation module specifically includes:

[0105] A center point coordinate calculation unit, used to calculate the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud;

[0106] A covariance matrix calculation unit, used to calculate the covariance matrix between the template point cloud and the marker point cloud based on the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud;

[0107] The rotation and translation matrix calculation unit is used to calculate the rotation and translation matrix based on the covariance matrix by using the SVD decomposition method.

[0108] The present invention solves the existing problem of the airport scene that only relies on GPS and scene radar for single location information for target positioning, and cannot locate the key points of the real spatial position of the target object. The present invention combines computer vision image processing methods with depth information and point cloud information to match the complete and accurate template point cloud information to the photographed target point cloud, and further locates the target object in the target space, making up for the current problem of single target positioning information and poor monitoring effect.

[0109] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0110] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for locating an airport surface target, characterized in that: include: Build a point cloud database; The point cloud database includes template point clouds of different targets of various aircraft types at the airport scene; The visible light images of various aircraft types collected in real time at the airport are detected by a trained landmark detection model to obtain landmark information of targets of different categories; the training process of the landmark detection model is as follows: collecting visible light images of various aircraft types at the airport; marking the targets and landmarks of the targets in the visible light images to construct a training set; training the landmark detection model by the training set to obtain a trained landmark detection model; Based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time, the marker point information is restored into a marker point cloud; Matching the template point cloud with the marker point cloud, and calculating the rotation and translation matrix; The rotation and translation matrix includes a rotation matrix and a translation matrix; The coordinates of the target object positioning point are calculated based on the rotation and translation matrix.

2. The method for locating an airport target according to claim 1, characterized in that: Based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time, the marker point information of the different categories of targets is restored into a marker point cloud, specifically including: Align the depth image with the visible light image, take the RGB camera coordinate system as the reference coordinate system, and extract the depth value of the marker point; The marker point information is restored into a marker point cloud based on the depth value and the camera projection model.

3. The method for locating an airport target according to claim 1, characterized in that: Matching the template point cloud with the marker point cloud and calculating the rotation and translation matrix specifically includes: Calculate the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud; Calculating a covariance matrix between the template point cloud and the marker point cloud based on the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud; Based on the covariance matrix, the rotation and translation matrix is ​​calculated using the SVD decomposition method.

4. An airport surface target positioning system, characterized in that: include: Point cloud database construction module, used to build point cloud database; The point cloud database includes template point clouds of different targets of various aircraft types at the airport scene; The landmark information detection module is used to detect the visible light images of various aircraft types collected in real time at the airport scene through the trained landmark detection model to obtain the landmark information of different categories of targets; A marker point cloud recovery module is used to restore the marker point information into a marker point cloud based on the depth image corresponding to the visible light image of each aircraft type at the airport scene collected in real time; A rotation and translation matrix calculation module, used for matching the template point cloud with the marker point cloud and calculating the rotation and translation matrix; The rotation and translation matrix includes a rotation matrix and a translation matrix; A target object positioning point coordinate calculation module, used to calculate the coordinates of the target object positioning point based on the rotation and translation matrix; It also includes: a training module, which specifically includes: Visible light image acquisition unit, used to collect visible light images of various aircraft types at the airport; A training set construction unit, used to mark the target object and the landmark points of the target object in the visible light image to construct a training set; The training unit is used to train the landmark detection model using the training set to obtain a trained landmark detection model.

5. The airport surface target positioning system according to claim 4, characterized in that: The landmark point cloud recovery module specifically includes: A depth value extraction unit, used to align the depth image with the visible light image, and extract the depth value of the marker point using the RGB camera coordinate system as a reference coordinate system; A restoration unit is used to restore the marker point information into a marker point cloud based on the depth value and the camera projection model.

6. The airport surface target positioning system according to claim 4, characterized in that: The rotation and translation matrix calculation module specifically includes: A center point coordinate calculation unit, used to calculate the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud; A covariance matrix calculation unit, used to calculate the covariance matrix between the template point cloud and the marker point cloud based on the center point coordinates of the template point cloud and the center point coordinates of the marker point cloud; The rotation and translation matrix calculation unit is used to calculate the rotation and translation matrix based on the covariance matrix by using the SVD decomposition method.

Images