Parking apron wingtip conflict real-time detection method and system based on deep learning

Abstract

The invention discloses a parking apron wingtip conflict real-time detection method and system based on deep learning, and relates to the technical field of parking apron safety, and the method comprises the steps: collecting a plurality of original sample images of different models of airplanes, marking key points, carrying out the preprocessing of the marked sample images, and dividing the preprocessed sample images into a training set and a verification set; constructing an initial aircraft detection model based on a YOLO algorithm, and training by using the training set and the verification set until the loss function is converged to obtain a trained aircraft detection model; inputting a to-be-recognized video acquired by a certain airplane into the trained airplane detection model to obtain a detection frame of a target airplane on each frame of image; and tracking the trajectory of the target aircraft in the detection frame, judging whether the trajectory of the aircraft coincides with the trajectory of the target aircraft, and if yes, sending alarm information to an airborne alarm system of the aircraft. According to the method, whether potential wingtip accidents exist or not can be quickly and accurately judged, and the safety of movement of aircrafts on the parking apron is improved.

Description

Technical Field

[0001] The present invention relates to the field of apron safety technology, and more specifically to a real-time detection method and system for apron wingtip conflicts based on deep learning. Background Art

[0002] The apron is a place for aircraft parking, take-off and landing, and ground services. It is usually located next to the airport runway, providing a flat road surface for aircraft to ensure the safety of aircraft during take-off, landing and taxiing. As a harbor for aircraft take-off and landing, it is not only a place for aircraft to stay for a short time, but also a key node to ensure the smooth progress of aviation activities. Aviation safety is an indispensable topic on every flight day. With the continuous development of the aviation industry, the apron is also facing new challenges and opportunities. It needs to constantly adapt to new needs and changes to promote the sustainable development of the aviation industry.

[0003] There are many challenges and risks in the operation of the airport apron, and crowded aprons are prone to aircraft scratch accidents. The wingtip is the outermost edge of the aircraft wing, usually the tapered tip of the wing, which is of great significance to the performance, safety and passenger comfort of the aircraft. During takeoff, the wingtip can help the take-off aircraft obtain better lift. During flight, the wingtip can provide the aircraft with better stability and aerodynamic performance. During landing, the wingtip assists the aircraft in landing smoothly by increasing the lift of the wing. Therefore, if the wingtip is damaged due to technical failure, human negligence, meteorological conditions, etc., it will cause other flights to be delayed or cancelled, resulting in huge property losses, and in serious cases, casualties.

[0004] In order to improve the safety of aircraft activities on the tarmac, researchers have tried to predict potential wingtip accidents through some perception methods, but there are still some shortcomings in detection efficiency and accuracy, which is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a real-time detection method and system for apron wingtip conflicts based on deep learning, which solves the problems existing in the background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A real-time detection method for apron wingtip conflicts based on deep learning, comprising the following steps:

[0008] S1. Collect several original sample images of different aircraft models, mark the key points of the original sample images, and add sample labels;

[0009] S2, preprocessing the labeled sample images, and dividing the preprocessed sample images into a training set and a verification set according to a preset ratio;

[0010] S3. Build an improved YOLO network architecture based on the YOLO algorithm to obtain the initial aircraft detection model;

[0011] S4, inputting the training set and the validation set into the initial aircraft detection model for model training until the loss function converges, thereby obtaining a trained aircraft detection model;

[0012] S5, inputting the to-be-identified video collected from a certain aircraft on the apron into the trained aircraft detection model, and obtaining a detection frame of the target aircraft on each frame of the to-be-identified video;

[0013] S6. Track the trajectory of the target aircraft in the detection frame to determine whether the trajectory of the own aircraft overlaps with that of the target aircraft. If there is overlap, immediately send an alarm message to the airborne alarm system of the own aircraft.

[0014] Optionally, the specific operation of S1 is:

[0015] Take photos of different types of aircraft at different shooting angles and distances to obtain several original sample images;

[0016] Based on different aircraft models, the locations in the aircraft structure that are most likely to cause collision risks and can cover the entire outline of the aircraft are selected as key points to determine the aircraft's area of ​​interest;

[0017] Based on the region of interest, key points are annotated on the original sample image and sample labels are added.

[0018] Optionally, the specific operation of S2 is:

[0019] Perform grayscale processing and smoothing filtering on the labeled sample image to obtain an optimized image;

[0020] The optimized image is subjected to background suppression and binarization processing, and the aircraft target area is extracted from the binarized image to complete the image preprocessing.

[0021] Optionally, the method further includes:

[0022] Perform left-right mirror flipping, top-bottom mirror flipping and rotation operations on each preprocessed sample image with sample labels to obtain an expanded image;

[0023] The data set consisting of the expanded images and the preprocessed sample images is divided according to a preset ratio to obtain a training set and a validation set.

[0024] Optionally, in S3, the initial aircraft detection model is built based on YOLOv5, including: input layer, backbone network, neck network and output layer;

[0025] The input layer is used to receive sample images input into the model and perform data augmentation;

[0026] The backbone network uses the MobileNet network to extract image features;

[0027] The neck network adopts the FPN_PAN structure and embeds the attention mechanism. The attention mechanism includes a channel attention module and a spatial attention module to improve the ability to extract key features of the image and describe the image features from the global receptive field.

[0028] The output layer uses SIoU as the loss function to correct the spatial orientation factor between the real bounding box of the training set and the predicted bounding box output by the model, and outputs the target detection box.

[0029] Optionally, the MobileNet network includes: a convolution layer, a plurality of depth-separable convolution layers, a global average pooling layer, and a fully connected layer; wherein the depth-separable convolution layer includes a depth convolution kernel and a point convolution kernel;

[0030] The convolution layer is used to perform edge compensation and convolution operations on the input initial image, and then obtain the feature map through BN processing and ReLU activation function;

[0031] The global average pooling layer is used to average pool the feature map output by the last depth-separable convolutional layer so that the final output feature map size is 1×1;

[0032] The number of neurons in the fully connected layer is the number of key points in the image, which are classified by the Softmax classifier.

[0033] Optionally, the specific operation of S4 is:

[0034] Initialize model parameters and input the training set into the initial aircraft detection model in batches for model training;

[0035] During the model training process, the validation set is used to adjust the model parameters through the small batch stochastic gradient descent method and the Adam optimizer. When the minimization loss error of the validation set reaches the minimum value, the model training is completed.

[0036] Optionally, the specific operation of S6 is:

[0037] Add the first frame of the image with the detection frame in the video to be identified to an empty trajectory chain, and traverse all the images in the video to be identified in sequence;

[0038] According to the last frame of each trajectory chain, the prediction box of the target aircraft on the next frame is predicted and matched with the detection box on the currently traversed picture, and the obtained trajectory chain of the target aircraft is determined as the motion trajectory of the target aircraft;

[0039] The degree of overlap between the target aircraft's trajectory and the aircraft's trajectory is obtained. If the overlap exceeds a preset threshold, an alarm message is immediately sent through the airborne warning system.

[0040] A real-time detection system for a wingtip collision on a ramp based on deep learning, using the real-time detection method for a wingtip collision on a ramp based on deep learning as described in any one of the above items, comprising:

[0041] An acquisition module is used to collect several original sample images of different aircraft models;

[0042] The annotation module is used to annotate the key points of the original sample image and add sample labels;

[0043] A preprocessing module is used to preprocess the labeled sample images and divide the preprocessed sample images into a training set and a verification set according to a preset ratio;

[0044] The model building module is used to build an improved YOLO network architecture based on the YOLO algorithm to obtain the initial aircraft detection model;

[0045] The model training module is used to input the training set and the validation set into the initial aircraft detection model for model training until the loss function converges to obtain a trained aircraft detection model;

[0046] The detection module is used to input the video to be identified collected from a certain aircraft on the apron into the trained aircraft detection model to obtain the detection frame of the target aircraft on each frame of the video to be identified;

[0047] The judgment module is used to track the trajectory of the target aircraft in the detection frame and determine whether the trajectory of the aircraft overlaps with that of the target aircraft. If there is overlap, an alarm message is immediately sent to the onboard warning system of the aircraft.

[0048] It can be seen from the above technical solutions that compared with the prior art, the present invention discloses a real-time detection method and system for wingtip conflicts on the apron based on deep learning, adds an attention mechanism to the YOLO algorithm, constructs an improved YOLO network architecture, and uses collected images of aircraft of different models for model training. More effective key features of the image can be extracted, the accuracy and efficiency of detection can be improved, and it can be more quickly and accurately determined whether there is a potential wingtip accident and remind the staff to take relevant measures in time, thereby improving the safety of aircraft activities on the apron. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 A flow chart of a method for real-time detection of wingtip conflicts on a ramp based on deep learning provided by the present invention;

[0051] Figure 2 This is a structural diagram of the deep learning-based real-time detection system for apron wingtip conflicts provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The embodiment of the present invention discloses a real-time detection method of a wingtip collision on a ramp based on deep learning, such as Figure 1 As shown, the following steps are included:

[0054] S1. Collect several original sample images of different aircraft models, mark the key points of the original sample images, and add sample labels;

[0055] S2, preprocessing the labeled sample images, and dividing the preprocessed sample images into a training set and a verification set according to a preset ratio;

[0056] S3. Build an improved YOLO network architecture based on the YOLO algorithm to obtain the initial aircraft detection model;

[0057] S4, inputting the training set and the validation set into the initial aircraft detection model for model training until the loss function converges, thereby obtaining a trained aircraft detection model;

[0058] S5, inputting the to-be-identified video collected from a certain aircraft on the apron into the trained aircraft detection model, and obtaining a detection frame of the target aircraft on each frame of the to-be-identified video;

[0059] S6. Track the trajectory of the target aircraft in the detection frame to determine whether the trajectory of the own aircraft overlaps with that of the target aircraft. If there is overlap, immediately send an alarm message to the airborne alarm system of the own aircraft.

[0060] Furthermore, in this embodiment, the specific operation of S1 is:

[0061] Take photos of different types of aircraft at different shooting angles and distances to obtain several original sample images;

[0062] Based on different aircraft models, the locations in the aircraft structure that are most likely to cause collision risks and can cover the entire outline of the aircraft are selected as key points to determine the aircraft's area of ​​interest;

[0063] Based on the region of interest, key points are annotated on the original sample image and sample labels are added.

[0064] Furthermore, in this embodiment, the specific operation of S2 is:

[0065] Perform grayscale processing and smoothing filtering on the labeled sample image to obtain an optimized image;

[0066] The optimized image is subjected to background suppression and binarization processing, and the aircraft target area is extracted from the binarized image to complete the image preprocessing.

[0067] Specifically, the labeled sample image is converted into a grayscale image and smoothed and filtered to reduce the interference of noise in the image and facilitate post-processing; the optimized image is subjected to background suppression and binarization by means of an adaptive threshold; the binary image is subjected to connected domain detection and the area and centroid information of the connected domain are extracted; the connected domain with an area greater than a given value and the smallest centroid coordinate is taken as the aircraft target area, and the preprocessed sample image is obtained.

[0068] Furthermore, in another embodiment, the method further comprises:

[0069] Perform left-right mirror flipping, top-bottom mirror flipping and rotation operations on each preprocessed sample image with sample labels to obtain an expanded image;

[0070] The data set consisting of the expanded images and the preprocessed sample images is divided according to a preset ratio to obtain a training set and a validation set.

[0071] Specifically, the sample image is rotated, that is, the center of the sample image is used as the origin, and the label of the sample image and the sample image itself are rotated by a certain angle relative to the origin. In this embodiment, by expanding the sample image, more reference data can be provided for subsequent model training, thereby enhancing the model's ability to detect aircraft.

[0072] Further, in this embodiment, in S3, the initial aircraft detection model is built based on YOLOv5, including: an input layer, a backbone network, a neck network, and an output layer;

[0073] The input layer is used to receive sample images input into the model and perform data augmentation;

[0074] The backbone network uses the MobileNet network to extract image features;

[0075] The neck network adopts the FPN_PAN structure and embeds the attention mechanism. The attention mechanism includes a channel attention module and a spatial attention module to improve the ability to extract key features of the image and describe the image features from the global receptive field.

[0076] The output layer uses SIoU as the loss function to correct the spatial orientation factor between the real bounding box of the training set and the predicted bounding box output by the model, and outputs the target detection box.

[0077] Specifically, the introduction of the attention mechanism based on the YOLOv5 architecture can improve the network's ability to extract complex information features, that is, it can improve the ability to extract key features of aircraft images, so that the model can more clearly recognize the differences between different structures, which is helpful for the detection of target aircraft. Among them, the neck network is constructed based on the FPN_PAN structure. The horizontal axis of the FPN feature pyramid structure is regarded as the scale axis, and the scale-invariant feature variables are extracted. The pyramid feature map is evenly adjusted to a high-resolution feature pyramid map and connected with the extracted scale-invariant feature variables, which are used for the YOLO model to detect the feature area of ​​the aircraft.

[0078] The calculation formula of SIoU is:

[0079]

[0080] Where: λ represents the distance between the center point of the predicted bounding box and the center point of the true bounding box, as well as the angle formed by the connecting line and the vertical line of the two points; ζ represents the similarity of the shape of the predicted bounding box and the true bounding box; IoU represents the IoU value between the predicted bounding box and the true bounding box.

[0081] Furthermore, the MobileNet network includes: a convolutional layer, several depth-wise separable convolutional layers, a global average pooling layer, and a fully connected layer; wherein the depth-wise separable convolutional layer includes depth-wise convolutional kernels and point-wise convolutional kernels;

[0082] The convolution layer is used to perform edge compensation and convolution operations on the input initial image, and then obtain the feature map through BN processing and ReLU activation function;

[0083] The global average pooling layer is used to average pool the feature map output by the last depth-separable convolutional layer so that the final output feature map size is 1×1;

[0084] The number of neurons in the fully connected layer is the number of key points in the image, which are classified by the Softmax classifier.

[0085] Furthermore, in this embodiment, the specific operation of S4 is:

[0086] Initialize model parameters and input the training set into the initial aircraft detection model in batches for model training;

[0087] During the model training process, the validation set is used to adjust the model parameters through the small batch stochastic gradient descent method and the Adam optimizer. When the minimization loss error of the validation set reaches the minimum value, the model training is completed.

[0088] Specifically, the model is trained by the training set and the validation set, and the model parameters that minimize the minimum loss error of the validation set are saved as the optimal parameters of the aircraft detection model. When the minimum loss error of the validation set recorded continuously begins to become larger than the minimum value recorded previously, the model training stops. Through this method, the present embodiment can obtain an aircraft detection model that meets the requirements more, improve the accuracy of model prediction, and timely discover target aircraft that may have a collision risk to avoid causing greater losses.

[0089] Furthermore, in this embodiment, the specific operation of S6 is:

[0090] Add the first frame of the image with the detection frame in the video to be identified to an empty trajectory chain, and traverse all the images in the video to be identified in sequence;

[0091] According to the last frame of each trajectory chain, the prediction box of the target aircraft on the next frame is predicted and matched with the detection box on the currently traversed picture, and the obtained trajectory chain of the target aircraft is determined as the motion trajectory of the target aircraft;

[0092] The degree of overlap between the target aircraft's trajectory and the aircraft's trajectory is obtained. If the overlap exceeds a preset threshold, an alarm message is immediately sent through the airborne warning system.

[0093] Specifically, the Kalman filter algorithm can be used to predict the prediction box of the target aircraft on the next frame of the picture based on the last frame of the picture; thereafter, when matching the prediction box on the next frame of the picture with the detection box on the currently traversed picture, it is determined whether the number of consecutive labeled pictures in the trajectory chain exceeds the preset number. If it does not exceed the preset number, IOU matching is performed, which can comprehensively consider the overlap, center point distance, and aspect ratio consistency of the predicted bounding box and the real bounding box, thereby improving the accuracy of positioning the target aircraft; if it exceeds the preset number, the prediction box is cascade matched with each traversed detection box, and the motion trajectory of the target aircraft is obtained according to the traversed trajectory chain.

[0094] and Figure 1 Corresponding to the method described above, the embodiment of the present invention also provides a real-time detection system for wingtip collision on a ramp based on deep learning, which is used to Figure 1 In the specific implementation of the method, a real-time detection system for wingtip collision on a ramp based on deep learning provided by an embodiment of the present invention can be applied to a computer terminal or various mobile devices, such as Figure 2 As shown, specifically including:

[0095] An acquisition module is used to collect several original sample images of different aircraft models;

[0096] The annotation module is used to annotate the key points of the original sample image and add sample labels;

[0097] A preprocessing module is used to preprocess the labeled sample images and divide the preprocessed sample images into a training set and a verification set according to a preset ratio;

[0098] The model building module is used to build an improved YOLO network architecture based on the YOLO algorithm to obtain the initial aircraft detection model;

[0099] The model training module is used to input the training set and the validation set into the initial aircraft detection model for model training until the loss function converges to obtain a trained aircraft detection model;

[0100] The detection module is used to input the video to be identified collected from a certain aircraft on the apron into the trained aircraft detection model to obtain the detection frame of the target aircraft on each frame of the video to be identified;

[0101] The judgment module is used to track the trajectory of the target aircraft in the detection frame and determine whether the trajectory of the aircraft overlaps with that of the target aircraft. If there is overlap, an alarm message is immediately sent to the onboard warning system of the aircraft.

[0102] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0103] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A real-time detection method for apron wingtip conflicts based on deep learning, characterized in that: The following steps are involved: S1. Collect several original sample images of different aircraft models, mark the key points of the original sample images, and add sample labels; S2, preprocessing the labeled sample images, and dividing the preprocessed sample images into a training set and a verification set according to a preset ratio; S3. Build an improved YOLO network architecture based on the YOLO algorithm to obtain the initial aircraft detection model; S4, inputting the training set and the validation set into the initial aircraft detection model for model training until the loss function converges, thereby obtaining a trained aircraft detection model; S5, inputting the to-be-identified video collected from a certain aircraft on the apron into the trained aircraft detection model, and obtaining a detection frame of the target aircraft on each frame of the to-be-identified video; S6. Track the trajectory of the target aircraft in the detection frame to determine whether the trajectory of the own aircraft overlaps with that of the target aircraft. If there is overlap, immediately send an alarm message to the airborne alarm system of the own aircraft.

2. The method for real-time detection of wingtip collisions on a ramp based on deep learning according to claim 1, characterized in that: The specific operations of S1 are: Take photos of different types of aircraft at different shooting angles and distances to obtain several original sample images; Based on different aircraft models, the locations in the aircraft structure that are most likely to cause collision risks and can cover the entire outline of the aircraft are selected as key points to determine the aircraft's area of ​​interest; Based on the region of interest, key points are annotated on the original sample image and sample labels are added.

3. The method for real-time detection of wingtip collision on apron based on deep learning according to claim 1, characterized in that: The specific operations of S2 are: Perform grayscale processing and smoothing filtering on the labeled sample image to obtain an optimized image; The optimized image is subjected to background suppression and binarization processing, and the aircraft target area is extracted from the binarized image to complete the image preprocessing.

4. The method for real-time detection of wingtip collision on a ramp based on deep learning according to claim 1, characterized in that: Also includes: Each pre-processed sample image with a sample label is subjected to left-right mirror flipping, top-bottom mirror flipping and rotation operations to obtain an expanded image; The data set consisting of the expanded images and the preprocessed sample images is divided according to a preset ratio to obtain a training set and a validation set.

5. The method for real-time detection of apron wingtip conflicts based on deep learning according to claim 1, characterized in that: In S3, the initial aircraft detection model is built based on YOLOv5, including: input layer, backbone network, neck network and output layer; The input layer is used to receive sample images input into the model and perform data augmentation; The backbone network uses the MobileNet network to extract image features; The neck network adopts the FPN_PAN structure and embeds the attention mechanism. The attention mechanism includes a channel attention module and a spatial attention module to improve the ability to extract key features of the image and describe the image features from the global receptive field. The output layer uses SIoU as the loss function to correct the spatial orientation factor between the real bounding box of the training set and the predicted bounding box output by the model, and outputs the target detection box.

6. The method for real-time detection of wingtip conflicts on a ramp based on deep learning according to claim 5, characterized in that: The MobileNet network includes: a convolutional layer, several depth-wise separable convolutional layers, a global average pooling layer, and a fully connected layer; the depth-wise separable convolutional layer includes depth-wise convolution kernels and point-wise convolution kernels; The convolution layer is used to perform edge compensation and convolution operations on the input initial image, and then obtain the feature map through BN processing and ReLU activation function; The global average pooling layer is used to average pool the feature map output by the last depth-separable convolutional layer so that the final output feature map size is 1×1; The number of neurons in the fully connected layer is the number of key points in the image, which are classified by the Softmax classifier.

7. The method for real-time detection of wingtip collision on a ramp based on deep learning according to claim 1, characterized in that: The specific operations of S4 are: Initialize model parameters and input the training set into the initial aircraft detection model in batches for model training; During the model training process, the validation set is used to adjust the model parameters through the small batch stochastic gradient descent method and the Adam optimizer. When the minimization loss error of the validation set reaches the minimum value, the model training is completed.

8. The method for real-time detection of apron wingtip conflicts based on deep learning according to claim 1, characterized in that: The specific operations of S6 are: Add the first frame of the image with the detection frame in the video to be identified to an empty trajectory chain, and traverse all the images in the video to be identified in sequence; According to the last frame of each trajectory chain, the prediction box of the target aircraft on the next frame is predicted and matched with the detection box on the currently traversed picture, and the obtained trajectory chain of the target aircraft is determined as the motion trajectory of the target aircraft; The degree of overlap between the target aircraft's trajectory and the aircraft's trajectory is obtained. If the overlap exceeds a preset threshold, an alarm message is immediately sent through the airborne warning system.

9. A real-time detection system for apron wingtip conflicts based on deep learning, using the real-time detection method for apron wingtip conflicts based on deep learning as described in any one of claims 1 to 8, characterized in that: include: An acquisition module is used to collect several original sample images of different aircraft models; The annotation module is used to annotate the key points of the original sample image and add sample labels; A preprocessing module is used to preprocess the labeled sample images and divide the preprocessed sample images into a training set and a verification set according to a preset ratio; The model building module is used to build an improved YOLO network architecture based on the YOLO algorithm to obtain the initial aircraft detection model; The model training module is used to input the training set and the validation set into the initial aircraft detection model for model training until the loss function converges to obtain a trained aircraft detection model; The detection module is used to input the video to be identified collected from a certain aircraft on the apron into the trained aircraft detection model to obtain the detection frame of the target aircraft on each frame of the video to be identified; The judgment module is used to track the trajectory of the target aircraft in the detection frame and determine whether the trajectory of the aircraft overlaps with that of the target aircraft. If there is overlap, an alarm message is immediately sent to the onboard warning system of the aircraft.

Images