An airport parking space intrusion detection method based on deep learning

Abstract

The application discloses a kind of detection methods of airport parking space invasion based on deep learning, comprising the following steps: obtaining video stream, the region of parking space is drawn in image;Carry out aircraft target detection in parking space region, judge whether there is aircraft in parking space;If no aircraft is detected, start human body and special vehicle detection module, judge whether staff and special vehicle are in parking space range, if exist, system carries out invasion alarm;If aircraft is detected in parking space and is in berth or off-site state, start human body and special vehicle detection module, such as berth state human body and special vehicle are in parking space range, or off-site state, other targets are in parking space range, system carries out invasion alarm;If aircraft is in parking state, do not make further processing.The application fully considers aircraft berth and off-site situation, introduces multi-scale target detection, the accuracy of detection is high, generalization ability is strong, meets the use requirement of actual business scene.

Description

[TECHNICAL FIELD]

[0001] The application relates to airport parking space management, in particular to an airport parking space intrusion detection method based on deep learning. [BACKGROUND]

[0002] Air transportation has the characteristics of rapidity, long distance, comfort and safety, and has irreplaceable advantages in the modern transportation system, and plays an increasingly important role in politics, economy, culture and social construction. With the rapid development of China's aviation transportation industry, how to efficiently and strictly implement the supervision of airport operations is becoming more and more important.

[0003] The parking space is the position of the aircraft parking, and is an important part of the ground support, so it is one of the important work to ensure the safety of the parking space and the detection of the aircraft parking safety. According to the civil aviation regulations, the following situations are not allowed to enter the parking area illegally: first, there is no aircraft parking in the parking space range, and no personnel and special vehicles are allowed to enter; second, when the aircraft is in the parking space (the front wheels of the aircraft have not parked on the designated parking line), no personnel and special vehicles are allowed to enter; third, when the aircraft starts to leave the parking space, no target is allowed to enter except the personnel and the aircraft towing vehicle.

[0004] When the aircraft parks to the preset position, the related operation personnel and special vehicles can be allowed to enter, which are responsible for the aircraft guiding, the corridor bridge docking, the luggage transportation, the catering transportation, the cleaning, the safety detection, the refueling and the charging and the like. The environment of the parking apron is divided into day, night, rainy day and sunny day. In the traditional detection mode, the real-time video is transmitted back through the on-site camera, and the human eye is used to judge whether the parking space exists the intrusion condition, which needs to spend high human cost and time cost.

[0005] The application No. CN202010326296.1 discloses an airport parking space intrusion detection method and system. The detection method comprises the following steps: real-time acquisition of a monitoring image, and calculation of a perspective transformation matrix of a front view perspective angle and a parking space overhead view perspective angle; judgment of whether an airplane enters the parking space in the monitoring image; detection of the monitoring image in which the airplane does not enter the parking space, to obtain a target pixel point set and a target label matrix; transformation of the matrix to obtain a transformed matrix of the overhead view perspective angle; judgment of whether all the pixels of each target in the transformed matrix enter the parking space area, and if all the pixels enter the parking space area, it is judged that there is an intrusion in the parking space. The application is based on a yolov3 model, detects targets in the field to obtain target matrix information, then transforms the matrix to obtain a transformed matrix of the overhead view perspective angle, and then judges whether each target in the transformed matrix enters the parking space area. The application has the following disadvantages: the yolov3 detection model has low detection accuracy and slow detection speed; the application is only applicable to the case that there is no airplane in the parking space, without considering the cases that the airplane is parked or deparked; and the application does not consider the influence of complex environments (day, night, sunny day, rainy day), and the model has low robustness and low accuracy and recall rate in complex weather. [SUMMARY]

[0006] The technical problem to be solved by the application is to provide an airport parking space intrusion detection method with high detection accuracy.

[0007] To solve the above technical problem, the application adopts the technical solution of an airport parking space intrusion detection method, which comprises the following steps:

[0008] 101) obtaining a video stream from a field camera;

[0009] 102) drawing a parking space area in an image of the video;

[0010] 103) performing target detection of an airplane in the image parking space area, judging whether the parking space has an airplane parked, and if necessary, performing target detection of a person and a vehicle;

[0011] 104) if no airplane target is detected, starting a human body and special vehicle detection module to obtain position information of a staff and a special vehicle target in the image, judging whether the target is in the parking space range according to the position information of the target, and if the target is in the parking space range, the system performs intrusion alarm;

[0012] 105) if the airplane target is detected in the parking space, judging whether the airplane is parked, deparked or parked on a specified parking line;

[0013] 106) If the aircraft is in the berth state, start the human body and special vehicle detection module to obtain the position information of the staff and special vehicle targets in the image, and determine whether the target is in the range of the parking stand through the position information of the target. If the target exists, the system performs intrusion alarm;

[0014] 107) If the aircraft is in the off-site state, start the human body and special vehicle detection module to obtain the position information of the staff and special vehicle targets in the image, and determine whether there are other targets in the range of the parking stand through the target category information and the target position information, except for the off-site security personnel and the towing vehicle. If there are other targets in the range of the parking stand, the system performs intrusion alarm;

[0015] 108) If the aircraft is in the parking state, no further processing is performed.

[0016] The above-mentioned airport parking stand intrusion detection method, the target detection of step 103 includes the following steps:

[0017] 201) Collect the images of the aircraft and front wheels, staff, and special vehicles under the airport apron scene, and then label the data;

[0018] 202) Obtain the size of the prior box through the k-means clustering algorithm, and cluster the prior box into 9 different sizes;

[0019] 203) Configure the parameters of the yolov4 network model training, use the divided training set to iteratively train the pre-trained model, and save the trained model through the set training strategy;

[0020] 204) Perform aircraft target detection on the video frame through the aircraft detection model trained in step 203 to obtain a multidimensional array of target box position information and confidence information of the detected target;

[0021] 205) Determine whether there is an aircraft target on the parking stand through the target detection of step 204. If no aircraft target is detected, jump to step 206. If an aircraft target has been detected, jump to step 208.

[0022] 206) Perform target detection on the staff and special vehicles in the picture through the model trained in step 203 to obtain a multidimensional array of target box position information and confidence information of the detected target, and calculate the center point position of the target box through the coordinate information;

[0023] 207) Determine whether the center point of the target box is within the set parking stand area according to the ray algorithm. If a target is found within the parking stand range, an alarm is issued, and the target has intruded into the parking stand. Otherwise, no processing is performed;

[0024] 208) When the aircraft target in the parking position is detected, the aircraft front wheel target coordinate information identified according to the model and the parking line coordinate information set in the parking position region are used to determine whether the aircraft is in the parking position, off position or parked state; if it is the parked state, no processing is performed; if the aircraft is in the parking position, steps 206 and 207 are returned to perform operations; if the aircraft is in the off position, step 209 is performed;

[0025] 209) The vehicle target in the parking position is identified by setting the judgment logic condition through the special vehicle detection model, and the influence of the tractor is excluded; the influence of the staff is excluded by setting the judgment logic condition through the staff detection model; the staff target in the parking position is identified, the coordinate information of the target is returned, the position of the staff in the parking position is determined according to the personnel target coordinate information; and it is determined whether the personnel is the aircraft off position support personnel; if it is determined that the personnel is not the aircraft off position support personnel, steps 206 and 207 are returned to perform operations.

[0026] The airport parking position intrusion detection method described above, step 201 includes the following steps:

[0027] 301) Use multiple fixed cameras at different angles to take pictures in each parking apron scene, and manually label each image; each aircraft, aircraft front wheel, operator and special vehicle in each image has a corresponding detection box, and the detection box is marked as [(x1, y1), (x2, y2)], (x1, y1) is the left upper corner coordinate of the detection box, and (x2, y2) is the right lower corner coordinate of the detection box; the label types include aircraft, aircraft front wheel, reflective vest body, ordinary work clothes body, ordinary body and special vehicle;

[0028] 302) Use a data enhancement method to reduce the overfitting phenomenon of the network, and train a network with stronger generalization ability; data enhancement includes data synthesis, random cropping, flipping, color jitter, adding noise and / or rotation on data samples;

[0029] 303) Secondary processing is performed on the image data, and the picture is enhanced by the following operations: conversion to grayscale image-two-dimensional convolution-image array bit depth conversion, so that the edge texture of the target in the image is clear;

[0030] 304) Finally, the aircraft and front wheel, staff and special vehicle database is formed.

[0031] The airport parking space intrusion detection method described above, step 202 includes the following steps: K-means algorithm, that is, k-means clustering algorithm, which includes the following steps: pre-allocate data into K groups, then randomly select K objects as initial cluster centers, then calculate the distance between each object and each seed cluster center, and assign each object to the cluster center closest to it; the cluster center and the objects assigned to them represent a cluster, and the cluster center of each cluster is recalculated according to the existing objects in the cluster; this process will be repeated until the set termination condition is met; for a given data set X containing n d-dimensional data points and the number of classes K to be divided, the Euclidean distance is selected as the similarity index, and the clustering goal is to minimize the sum of squares of each class, that is, to minimize, and the formula is as follows:

[0032]

[0033] The airport parking space intrusion detection method described above, step 203 includes the following steps:

[0034] 501) The target detection model includes an aircraft detection model, a staff detection model, and a special vehicle detection model. The target of the aircraft detection model is an aircraft and an aircraft front wheel. The target of the staff detection model includes a reflective vest body, a body wearing ordinary work clothes, and an ordinary body. The target of the special vehicle detection model includes all vehicle types in the airport;

[0035] 502) The training process of the model includes first inputting the picture into the CSPDarknet53 backbone network for feature extraction to generate feature maps of different scales. The input format is: 608*608*3, and the format of the generated multi-scale feature map is: 76*76*256, 38*38*512, and 19*19*1024;

[0036] 503) The feature information of feature maps of different sizes is fused through the SPP+PAN structure;

[0037] 504) The feature information is input into the confidence predictor to obtain a confidence map. The format of the confidence map is 76*76*21, 38*38*21, and 19*19*21. Through the confidence map and the target graph real value information, the class loss, confidence loss, and position loss of each scale feature are calculated. The three types of loss values of the three scale feature maps are added together to obtain the total loss value of the model;

[0038] 501) Through the total loss function information, the network weights of the model are updated by using the random gradient descent method for back propagation.

[0039] The airport parking space intrusion detection method described above, step 207 includes the following steps:

[0040] 601)In the image, the shape of the parking space is depicted, the coordinate information of each pixel point of the parking space shape is obtained, and then the position of the parking space is depicted in the image by using the cvPolyLine function of opencv, and the contour of the parking space region is simplified as a polygon;

[0041] 602)Judge whether the center point of the target frame is in the parking space region by the ray method: according to the set of all points of the polygon, the maximum X coordinate maxlng, the minimum X coordinate minlng, the maximum Y coordinate maxlat and the minimum Y coordinate minlat of the polygon are selected, the X coordinate x of the center point is compared with the values of maxlng and minlng respectively, and the Y coordinate y of the center point is compared with the values of maxlat and minlat respectively, when x>maxlng or x<minlng or y>maxlat or y<minlat, it can be judged that the center point is outside the region, and the target is outside the region;

[0042] 603)When the above conditions are not met, the position of the center point is further judged, and whether the X coordinate x and the Y coordinate y of the center point exist the same point with the coordinate points xi and yi of the parking space polygon region is judged in a loop, if yes, it is judged that the center point coincides with the point of the polygon, and the target is in the region;

[0043] 604)Judge whether the two end points of any line segment of the polygon are respectively on the two sides of the ray, so as to judge whether the line segment intersects with the straight line where the ray is located; when the direction of the ray is along the center point horizontally to the left, the X coordinate of the intersection point of the straight line where the ray is located and the line segment is calculated by using the following formula:

[0044]

[0045] 605)If xeg=x, it is judged that the intersection point is the center point position, and it is considered that the target is in the region; if xseg<x, it is judged that the intersection point is on the left side of the center point, and the intersection point number is accumulated by 1; all the line segments of the polygon are judged in a loop, when the intersection point number is odd, it is judged that the center point is in the region, and it is considered that the target is in the parking space range; when the intersection point number is even, it is judged that the center point is not in the region, and it is considered that the target is not in the parking space range.

[0046] The above-mentioned airport parking space intrusion detection method, step 208 includes the following steps:

[0047] 701)The coordinate information of the aircraft target and the front wheel target of the aircraft is identified by the aircraft detection model, and the target center point information of the aircraft target and the front wheel target of the aircraft;

[0048] 702)By comparing the parking line area coordinate position with the target center point information, the Euclidean distance between the two points is calculated, according to the change of the Euclidean distance, if the distance is getting smaller and smaller, it can be judged that the aircraft is parking; if the distance is getting larger and larger, it can be judged that the aircraft is leaving the parking, then the staff and special vehicle detection module is started, and the subsequent steps are executed.

[0049] The airport parking space intrusion detection method fully considers the parking and leaving of the aircraft, introduces multi-scale target detection, has high detection accuracy and strong generalization ability, and meets the use requirements of the actual business scene. [SUMMARY]

[0050] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0051] Figure 1 It is a flow chart of the airport parking space intrusion detection method of the embodiment of the application.

[0052] Figure 2 It is a flow chart of judging whether the target is in the intrusion area.

[0053] Figure 3 It is a flow chart of model training.

[0054] Figure 4 It is a whole structure block diagram of the model.

[0055] Figure 5 It is one of the schematic diagrams of judging the target center point by the ray method.

[0056] Figure 6 It is the second schematic diagram of judging the target center point by the ray method.

[0057] Figure 7 It is a schematic diagram of calculating the X coordinate of the intersection point of the ray and the line segment. [DETAILED DESCRIPTION]

[0058] The embodiment of the application discloses an airport parking space intrusion detection method based on deep learning, and the flow chart is as shown in Figure 1 The main process includes the following steps:

[0059] 1. Obtain a video stream from a field camera.

[0060] 2. Draw the area of the parking space in the video image.

[0061] 3. Perform aircraft target detection in the image area, judge whether the aircraft is parked in the parking space, and if necessary, perform human and vehicle target detection.

[0062] 4. If no aircraft target is detected, start the human and special vehicle detection module to obtain the position information of the staff and special vehicle targets in the image, and determine whether the target is in the range of the parking position through the position information of the target. If the target exists, the system performs intrusion alarm.

[0063] 5. If the aircraft target is detected in the parking position, determine whether the aircraft is in the berth, off the berth or has stopped at the designated parking line.

[0064] 6. If the aircraft is in the berth state, start the human and special vehicle detection module to obtain the position information of the staff and special vehicle targets in the image, and determine whether the target is in the range of the parking position through the position information of the target. If the target exists, the system performs intrusion alarm.

[0065] 7. If the aircraft is in the off-berth state, start the human and special vehicle detection module to obtain the position information of the staff and special vehicle targets in the image, and determine whether there are other targets in the range of the parking position through the target category information and the target position information, except for the off-berth support personnel and the towing vehicle. If there are other targets in the range of the parking position, the system performs intrusion alarm.

[0066] 8. If the aircraft is in the parking state, no processing is required.

[0067] The airport parking position intrusion detection method based on deep learning of the embodiment of the application determines whether the target is in the intrusion area as shown in the flowchart. Figure 2 The information of the target frame (x1, y1, w, h) is obtained through target detection, and then the position coordinates of the center point of the target frame are calculated through the above coordinate information. Then, through the ray method, it is determined whether the center point of the target is in the region, so as to determine whether the detected target intrudes into the specified region. The specific steps are as follows:

[0068] S1: Collect images of aircrafts and front wheels, staff and special vehicles under the scene of the parking apron, and then label the data.

[0069] S2: Obtain the size of the prior frame through the k-means clustering algorithm, and cluster 9 sizes of prior frames according to different scales. Because the format of the last generated multi-scale feature map of the yolov4 network is: 76*76*256, 38*38*512 and 19*19*1024, and each feature map defines 3 prior frames, so there are 9 prior frames in total,

[0070] S3: Configure the parameters of the yolov4 network model training, use the divided training set to iteratively train the pre-trained model, and save the trained model through the set training strategy.

[0071] S4: The aircraft detection model trained in step S3 is used to detect the aircraft target in the video frame, and a multi-dimensional array of target frame position information and confidence information of the detected target is obtained.

[0072] S5: Whether the aircraft target exists in the parking position is determined through the target detection in step S4. If no aircraft target is detected, step S6 is executed. If the aircraft target is detected, step S8 is executed.

[0073] S6: The model trained in step S3 is used to detect the staff and special vehicle targets in the picture, and a multi-dimensional array of target frame position information and confidence information of the detected target is obtained. The center point position of the target frame is calculated through the coordinate information, and the coordinate information includes the xy coordinate information of the upper left corner of the rectangular target frame and the xy coordinate information of the lower right corner.

[0074] S7: According to the ray algorithm (odd-even rule method), whether the target center point is in the set parking position area is determined. If the target is found in the parking position range, an alarm is sent to warn that the target has invaded the parking position. Otherwise, no processing is required, and step S8 operation is not required.

[0075] S8: After detecting the aircraft target in the parking position, whether the aircraft is in the parking position, off position or has been parked is determined according to the aircraft front wheel target coordinate information identified by the model and the parking line coordinate information set in the parking position area. If it is in the parked state, no processing is required. If it is in the parking position, steps S6 and S7 are executed. If it is in the off position, step S9 is executed.

[0076] S9: The special vehicle detection model is used to set the judgment logic condition, identify the vehicle target in the parking position, and exclude the influence of the tractor. The staff detection model is used to set the judgment logic condition and exclude the influence of the staff. The staff target in the parking position is identified, and the coordinate information of the target is returned. Whether the staff is the aircraft off position support personnel is determined according to the staff target coordinate information. If the staff is not the aircraft off position support personnel, steps S6 and S7 are executed.

[0077] Detailed description of step S1:

[0078] (1) Use different angle fixed camera to shoot images in each apron scene (including day, night, rainy and sunny), manually label each image. Each aircraft, aircraft nose wheel, operator, special vehicle in each image has a corresponding detection box, and the detection box is marked as [(x1, y1), (x2, y2)], (x1, y1) represents the left upper corner coordinate of the detection box, and (x2, y2) represents the right lower corner coordinate of the detection box. Label the target type to be detected, including: aircraft, aircraft nose wheel, person wearing reflective vest, person wearing ordinary work clothes, ordinary person, special vehicle.

[0079] (2) In view of the insufficient amount of field data, data enhancement method is used to reduce the overfitting phenomenon of network, and network with stronger generalization ability is trained. Data enhancement includes data synthesis, random cropping, flipping, color jitter, adding noise, rotation and other methods.

[0080] (3) Because the aircraft nose wheel is a small target, and the field environment is complex (day, night, rainy and sunny), which increases the difficulty of small target recognition in harsh environment. In order to better identify the aircraft and aircraft nose wheel, the image data needs to be processed twice, and the picture enhancement is carried out through the following operations: gray scale image-2D convolution-image array bit depth conversion, this step is realized by using the function method of opencv, the gray scale image is realized by using cvtcolor function, 2D convolution is realized by using flip and filter2D function parallel operation, and image array bit depth conversion is realized by using convertScaleAbs function, so that the edge texture of the target in the image is clearer, which is helpful for small target detection.

[0081] (4) Finally, the multi-dimensional feature information database of aircraft and nose wheel, staff and special vehicle is formed.

[0082] Detailed description of step S2:

[0083] (1) K-means algorithm, that is, k-means clustering algorithm is an iterative solution of clustering analysis algorithm, the steps are as follows: the data is divided into K groups, then K objects are randomly selected as initial cluster centers, then the distance between each object and each seed cluster center is calculated, and each object is assigned to the cluster center closest to it. The cluster center and the objects assigned to them represent a cluster. Each sample is assigned, and the cluster center of the cluster is recalculated according to the existing objects in the cluster. This process will be repeated until a certain termination condition is met. For a given data set X containing n d-dimensional data points and the number of classes K to be divided, the Euclidean distance is selected as the similarity index, and the clustering objective is to minimize the sum of squares of each class, that is, to minimize the formula as follows:

[0084]

[0085] In the above formula, k is a parameter for dividing n data objects into k clusters; x is a sample, the number of which is n; u is a random centroid point of the k clusters. J is a data set, that is, the distance of the data object to the K points is calculated, and the nearest one is selected to form a set. The mean of the data objects in each set is recalculated, and the mean is taken as a new cluster center point. The new cluster center point already exists, and on this basis, the distance of the entire data object to the cluster center point is recalculated, the clustering is re-divided, the mean is recalculated, the clustering is re-divided, and the iteration is continued until all the data objects cannot be updated to other data sets.

[0086] Detailed description of step S3:

[0087] (1) The target detection model has three, one is an airplane detection model, the target is an airplane and an airplane front wheel; the second is a staff detection model, including targets such as reflective clothing, ordinary work clothes, and ordinary human bodies. The third is a special vehicle detection model, the target is all vehicle types in the airport.

[0088] (2) The model training process is shown in Figure 3 , first input the picture into the CSPDarknet53 backbone network for feature extraction, and finally generate feature maps of different scales, the input format is: 608*608*3, and the format of the generated multi-scale feature map is: 76*76*256, 38*38*512, 19*19*1024.

[0089] (3) Then the feature information of different size feature maps is fused through SPP+PAN structure.

[0090] (4) The features are input into the confidence predictor to obtain the confidence map (76*76*21, 38*38*21, 19*19*21). Through the confidence map and the target map real value information (the position and category information of the manually labeled target in the image), the class loss, confidence loss and position loss of each scale feature are calculated. The three types of loss value formulas of the feature maps of the three scales are added together to obtain the total loss value formula of the model.

[0091] (5) Finally, through the total loss value formula, the random gradient descent method is used for back propagation,

[0092] constantly updating the network weights of the model.

[0093] The overall structure of the model is shown in Figure 4YOLOv4 network structure can be divided into three parts: backbone network (Backbone), neck network (Neck) and head (Head). The backbone network is the backbone feature extraction network, CSPDarknet53 contains five residual modules with stacking times of 1, 2, 8, 8 and 4, as shown in the figure. After inputting a picture with a size of 608x608x3, it uses convolution to continuously extract features, and the width and height of the picture are continuously compressed while the number of channels is continuously expanded, finally three output feature layers are obtained. The neck network is also called the enhanced feature extraction network. YOLOv4 uses SPP module and PANet network as the neck network. SPP uses maximum pooling with different pooling kernel sizes of 13x13, 9x9, 5x5 and 1x1 to process the last output feature layer of CSPDarknet53, which widens the receptive field and enhances the feature extraction ability of the feature map, effectively preventing overfitting. YOLOv4 uses a better PAN network to replace it, which realizes more effective fusion of features. The head structure of YOLOv4 uses three scale outputs for detecting targets of different sizes. The head with a size of 1 / 8, 1 / 16 and 1 / 32 of the original input size detects large, medium and small targets respectively. The depth of the head structure represents the boundary box offset, confidence, class and prior box, and each scale output has three prior boxes of different sizes.

[0094] Detailed description of step S7:

[0095] (1) The shape of the parking space is depicted in the image, the coordinate information of each pixel point of the parking space shape is obtained, and then the cvPolyLine function of opencv is used to depict the position of the parking space in the image. The contour of the parking space region is simplified as a polygon, and the shape of the polygon is determined according to the actual position of the parking space in the camera.

[0096] (2) As shown in Figure 5 and Figure 6 , whether the target center point is in the parking space region is judged by the ray method: the ray method is to start from the judgment point and make a horizontal ray to the right (or to the left), calculate the intersection points of the ray and each edge of the polygon, if the number of intersection points is odd, the judgment point is located in the polygon, if the number of intersection points is even, the judgment point is located outside the polygon,

[0097] (3) According to the set of all points of the polygon, the maximum X coordinate maxlng, the minimum X coordinate minlng, the maximum Y coordinate maxlat and the minimum Y coordinate minlat of the polygon are screened out, the values of the X coordinate x of the center point and maxlng and minlng are compared respectively, the values of the Y coordinate y of the center point and maxlat and minlat are compared respectively, when x>maxlng or x<minlng or y>maxlat or y<minlat, it can be judged that the center point is outside the region, and the target is outside the region.

[0098] (4) When the above conditions cannot be met, the position of the center point is further judged, and whether the X coordinate x and the Y coordinate y of the center point exist the same point with the coordinate points xi and yi of the parking position polygon region is judged in a loop, if yes, it is judged that the center point coincides with the point of the polygon, and the target is in the region.

[0099] (5) Whether the two end points of the line segment of the polygon are respectively on the two sides of the ray is judged, as shown in the following formula: Figure 7 The coordinates of the line segment are (x1, y1), (x2, y2), if y>y1 and y<y2 or y>y2 and y<y1, it can be judged that the ray of the center point and the line segment exist intersection point. Because the direction of the ray is set to be along the center point horizontally to the left, it is needed to judge whether the intersection point of the ray and the line segment of the polygon is on the left side or the right side of the center point, and the x coordinate of the intersection point of the ray and the line segment is calculated by using the following formula:

[0100]

[0101] (In the formula, p_x and p_y are the xy coordinate values of the judgment point, and ex, ey, sx, sy are the coordinate values of the two vertices of the line segment)

[0102] (6) If xeg=x, it can be judged that the intersection point is the center point position, and it can be considered that the target is in the region; if xseg<x, it can be judged that the intersection point is on the left side of the center point, the intersection point number is accumulated by 1, and then all the line segments of the polygon are judged in a loop, when the intersection point number is odd, it can be judged that the center point is in the region, and it can be considered that the target is in the parking position range, when the intersection point is even, it can be judged that the center point is not in the region, and it can be considered that the target is not in the parking position range.

[0103] Detailed description of step S8:

[0104] (1) The center point position of the aircraft target frame and the aircraft front wheel target frame is recognized by the aircraft detection model, and the center information (x, y, w, h) of the target includes the target center point coordinates x, y, the width w and the height h of the rectangular frame.

[0105] (2) By comparing the center point coordinates of the parking line and the target center point coordinates, the Euclidean distance between the two points is calculated, according to the change of the Euclidean distance, if the distance is getting smaller and smaller, it can be judged that the airplane is parking; if the distance is getting larger and larger, it can be judged that the airplane is leaving the parking position, then the staff and special vehicle detection module is started, and the subsequent steps are executed.

[0106] The airport parking position intrusion detection method based on deep learning of the above embodiments has the following technical effects:

[0107] 1. Target detection for airplanes, front wheels, staff and special vehicles in different weather environments, without scene limitation and with high accuracy.

[0108] 2. Target detection for airplanes, front wheels, staff and special vehicles under different cameras, without manual threshold adjustment.

[0109] 3. Effective target detection can be performed after the target is blocked, whether the target invades the parking position can be judged, without manual intervention.

[0110] The airport parking position intrusion detection method based on deep learning of the above embodiments utilizes a large amount of data, trains through a powerful yolov4 target detection model, fully considers the airplane parking and leaving conditions, introduces multi-scale target detection, so that the robustness of the algorithm is higher, the generalization ability is stronger, and it is more in line with the use requirements of actual business scenarios.

Claims

1. A method for detecting airport stand intrusion based on deep learning, characterized in that, The method comprises the following steps: 101) obtaining a video stream from a live camera; 102) drawing a region of a parking stand in an image of the video; 103) performing target detection of an aircraft in the region of the image parking stand, determining whether the parking stand has an aircraft parked, and performing target detection of a person and a vehicle; 104) if no aircraft target is detected, starting a human body and special vehicle detection module to obtain position information of a staff member and a special vehicle target in the image, and determining whether the target is within the parking stand range based on the position information of the target, and if the target exists, the system issues an intrusion alarm; 105) if the parking stand detects an aircraft target, determining whether the aircraft is in a parking position, a departure position, or has been parked on a designated parking line; 106) if the aircraft is in a parking position, starting a human body and special vehicle detection module to obtain position information of a staff member and a special vehicle target in the image, and determining whether the target is within the parking stand range based on the position information of the target, and if the target exists, the system issues an intrusion alarm; 107) if the aircraft is in a departure position, starting a human body and special vehicle detection module to obtain position information of a staff member and a special vehicle target in the image, and determining whether there is another target within the parking stand range based on the target category information and the target position information, and if there is another target within the parking stand range, the system issues an intrusion alarm; 108) if the aircraft is in a parked state, no further processing is performed; The target detection of step 103 comprises the following steps: 103-1) collecting image data of an aircraft, front wheels, a staff member, and a special vehicle in a parking apron scene, and then labeling the data; 103-2) obtaining the size of a prior box through a k-means clustering algorithm, and clustering 9 different sizes of prior boxes according to different scales; 103-3) configuring parameters for training a yolov4 network model, using a divided training set to iteratively train a pre-trained model, and saving the trained model through a set training strategy; 103-4) performing aircraft target detection on a video frame through the aircraft detection model trained in step 103-3, and obtaining a multidimensional array of target box position information and confidence information of the detected target; 103-5) determining whether there is an aircraft target on the parking stand through the target detection of step 103-4, and if no aircraft target is detected, jumping to step 103-6, and if an aircraft target is detected, jumping to step 103-8; 103-6) performing target detection on a staff member and a special vehicle in the image through the model trained in step 103-3, obtaining a multidimensional array of target box position information and confidence information of the detected target, and calculating the center point position of the target box through the coordinate information; 103-7) determining whether the center point of the target box is within the set parking stand region according to a ray algorithm, and if a target is found within the parking stand range, an alarm is issued, and the target has intruded into the parking stand; otherwise, no processing is performed; 103-8) After detecting the aircraft target in the parking position, the aircraft front wheel target coordinate information identified according to the model and the parking line coordinate information set in the parking position area are used to determine whether the aircraft is in the parking position, off position or parked state; if it is in the parked state, no processing is performed; if it is determined that the aircraft is in the parking position, steps 103-6 and 103-7 are returned to perform operations; if the aircraft is in the off position, step 103-9 is performed; 103-9) The special vehicle detection model is used to set the judgment logic condition, identify the vehicle target in the parking position, and exclude the influence of the towing vehicle; the staff detection model is used to set the judgment logic condition and exclude the influence of the staff; the staff target in the parking position is identified, the coordinate information of the target is returned, the position of the staff in the parking position is determined according to the coordinate information of the staff target, and it is determined whether the staff is the aircraft off position support staff; if it is determined that the staff is not the aircraft off position support staff, steps 103-6 and 103-7 are returned to perform operations.

2. The method of detecting intrusion into an airport stand of claim 1, wherein, Step 103-1 includes the following steps: 201) Use multiple fixed cameras at different angles to take pictures in each parking apron scene, and manually label each image; each aircraft, aircraft front wheel, operator and special vehicle in each image has a corresponding detection box, and the detection box is marked as [(x1, y1), (x2, y2)], (x1, y1) is the left upper corner coordinate of the detection box, and (x2, y2) is the right lower corner coordinate of the detection box; the type of target to be detected is labeled, including aircraft, aircraft front wheel, reflective vest body, ordinary work clothes body, ordinary body and special vehicle; 202) Use data enhancement method to reduce network overfitting phenomenon and train network with stronger generalization ability; data enhancement includes data synthesis, random cropping, flipping, color jitter, adding noise and / or rotation on data samples; 203) Perform secondary processing on image data, and perform picture enhancement through the following operations: gray scale image-2D convolution-image array bit depth conversion, so that the edge texture of the target in the image is clear; 204) Finally, form an aircraft and front wheel, staff and special vehicle database.

3. The method of detecting intrusion into an airport stand of claim 1, wherein, Step 103-2 includes the following steps: K-means algorithm, that is, k-means clustering algorithm, which includes the following steps: first, divide the data into K groups, then randomly select K objects as initial cluster centers, then calculate the distance between each object and each seed cluster center, and assign each object to the cluster center closest to it; the cluster center and the objects assigned to them represent a cluster, and the cluster center is recalculated according to the existing objects in the cluster every time a sample is assigned; this process will be repeated until the set termination condition is met; for a given data set X containing n d-dimensional data points and the number of classes K to be divided, the Euclidean distance is selected as the similarity measure, and the clustering objective is to minimize the sum of squares of each cluster, that is, to minimize, and the formula is as follows: 。 4. The method of detecting intrusion into an airport stand of claim 1, wherein, Step 103-3 includes the following steps: 401) The target detection model includes an airplane detection model, a staff detection model, and a special vehicle detection model. The targets of the airplane detection model are airplanes and airplane front wheels. The targets of the staff detection model include reflective clothing bodies, ordinary work clothes bodies, and ordinary bodies. The targets of the special vehicle detection model include all vehicle types in the airport; 402) The training process of the model includes first inputting the picture into the CSPDarknet53 backbone network for feature extraction to generate feature maps of different scales. The input format is: 608*608*3, and the format of the generated multi-scale feature maps is: 76*76*256, 38*38*512, and 19*19*1024; 403) The feature information of feature maps of different sizes is fused through the SPP+PAN structure; 404) The feature information is input into the confidence predictor to obtain the confidence map. The format of the confidence map is 76*76*21, 38*38*21, and 19*19*21. Through the confidence map and the real value information of the target map, the class loss, confidence loss, and position loss of each scale feature are calculated. The three types of loss values of the feature maps of the three scales are added to obtain the total loss value of the model; 405) Through the total loss function information, the network weight of the model is updated by using the random gradient descent method for back propagation.

5. The method of detecting intrusion into an airport stand of claim 1, wherein, Step 103-7 includes the following steps: 501) The shape of the parking stand is depicted in the image, the coordinate information of each pixel point of the parking stand shape is obtained, and then the position of the parking stand is depicted in the image by using the cvPolyLine function of opencv. The contour of the parking stand region is simplified as a polygon; 502) The center point of the target frame is judged to be in the parking stand region by using the ray method: according to the set of all points of the polygon, the maximum X coordinate maxlng, the minimum X coordinate minlng, the maximum Y coordinate maxlat, and the minimum Y coordinate minlat of the polygon are selected. The X coordinate x of the center point is compared with the values of maxlng and minlng, and the Y coordinate y of the center point is compared with the values of maxlat and minlat, respectively. When x>maxlng or x<minlng or y>maxlat or y<minlat, it can be judged that the center point is outside the region, and the target is outside the region; 503) When the above conditions are not met, the position of the center point is further judged. Whether the X coordinate x and the Y coordinate y of the center point coincide with the coordinate points xi and yi of the parking stand polygon region is judged in a loop. If there is a coincidence, it is judged that the center point coincides with the points of the polygon, and the target is in the region; 504) Whether the two end points of any line segment of the polygon are respectively on the two sides of the ray is judged to determine whether the line segment intersects with the straight line where the ray is located. When the direction of the ray is along the center point horizontally to the left, the X coordinate of the intersection point of the straight line where the ray is located and the line segment is calculated by using the following formula: ； 505)If Xseg=x, it is determined that the intersection point is the center point position, and it is considered that the target is in the area; if Xseg<x, it is determined that the intersection point is on the left side of the center point, and the intersection point number is accumulated by 1; all the line segments of the polygon are looped to make the determination, when the intersection point number is odd, it can be determined that the center point is in the area, and it is considered that the target is in the range of the parking position; when the intersection point is even, it is determined that the center point is not in the area, and it is considered that the target is not in the range of the parking position.

6. The method of detecting intrusion into an airport stand of claim 1, wherein, Step 103-8 includes the following steps: 601) Through the aircraft detection model, the coordinate information of the aircraft target and the aircraft front wheel target is identified, and the target center point information of the aircraft target and the aircraft front wheel target is identified; 602) By comparing the parking line area coordinate position with the target center point information, the Euclidean distance between the two points is calculated, according to the change of the Euclidean distance, if the distance is getting smaller and smaller, it can be judged that the aircraft is parking; if the distance is getting larger and larger, it can be judged that the aircraft is leaving the parking position, then the staff and special vehicle detection module is started, and the subsequent steps are executed.

Images