A real-time detection method and system for apron wingtip conflicts based on deep learning

Abstract

The present invention discloses a real-time detection method and system for wingtip conflicts on the apron based on deep learning, which relates to the technical field of apron safety and includes: collecting a number of original sample images of different types of aircraft and annotating key points, preprocessing the annotated sample images and dividing them into a training set and a validation set; constructing an initial aircraft detection model based on the YOLO algorithm and training it using the training set and the validation set until the loss function converges to obtain a trained aircraft detection model; inputting a video to be recognized collected by a certain aircraft into the trained aircraft detection model to obtain a detection frame of the target aircraft on each frame of the image; tracking the trajectory of the target aircraft in the detection frame to determine whether the trajectories of the own aircraft and the target aircraft coincide, and if there is a coincidence, sending an alarm message to the on-board warning system of the own aircraft. The present invention can quickly and accurately determine whether there is a potential wingtip accident and improve the safety of aircraft activities on the apron.

Description

Technical Field

[0001] The present invention relates to the technical field of apron safety, and more particularly to a real-time detection method and system for apron wingtip conflicts based on deep learning. Background Art

[0002] An apron is a place for aircraft to park, take off and land, and for ground services. It is usually located beside the airport runway, providing a flat road surface for aircraft to ensure the safety of aircraft during takeoff, landing and taxiing. As a harbor for aircraft takeoff and landing, it is not only a place for aircraft to stay briefly, but also a key node to ensure the smooth progress of aviation activities. Aviation safety is an indispensable topic for every flight day. With the continuous development of the aviation industry, the apron also faces new challenges and opportunities, and needs to continuously adapt to new demands and changes to promote the sustainable development of the aviation industry.

[0003] There are a large number of challenges and risks in the operation of the airport apron. Scratching accidents of aircraft are likely to occur on the crowded apron. The wingtip is the outermost edge of the aircraft wing, usually the conical tip of the wing, which is of great significance to the performance, safety and passenger comfort of the aircraft. At takeoff, the wingtip can help the takeoff aircraft obtain better lift. During flight, the wingtip can provide better stability and aerodynamic performance for the aircraft. At landing, the wingtip assists the aircraft to land smoothly by increasing the lift of the wing. Therefore, if the wingtips are damaged due to aircraft collisions caused by technical failures, human negligence, meteorological conditions, etc., it will cause delays or cancellations of other flights, resulting in huge property losses and even casualties in severe cases.

[0004] To improve the safety of aircraft activities on the apron, researchers have tried to predict potential wingtip accidents through some sensing methods, but there are still some deficiencies in detection efficiency and accuracy, which are technical problems that need to be solved urgently by those skilled in the art at present. Summary of the Invention

[0005] In view of this, the present invention provides a real-time detection method and system for apron wingtip conflicts based on deep learning, which solves the problems existing in the background art.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A real-time detection method for apron wingtip conflicts based on deep learning, comprising the following steps:

[0008] S1. Collect a number of original sample images of different types of aircraft, label the key points of the original sample images, and add sample labels;

[0009] S2. Preprocess the labeled sample images, and divide the preprocessed sample images into a training set and a validation set according to a preset ratio;

[0010] S3. Build an improved YOLO network architecture based on the YOLO algorithm to obtain an initial aircraft detection model;

[0011] S4. Input the training set and the validation set into the initial aircraft detection model for model training until the loss function converges, and obtain a trained aircraft detection model;

[0012] S5. Input the video to be recognized collected from a certain aircraft on the apron into the trained aircraft detection model to obtain the detection frames of the target aircraft on each frame image of the video to be recognized;

[0013] S6. Track the trajectory of the target aircraft in the detection frame, and judge whether the trajectory of the own aircraft coincides with that of the target aircraft. If there is a coincidence, immediately send an alarm message to the on-board warning system of the own aircraft.

[0014] Optionally, the specific operation of S1 is:

[0015] Take pictures of different models of aircraft at different shooting angles and distances to obtain a number of original sample images;

[0016] Based on the different aircraft models, select the position in the aircraft structure that is most likely to have a collision risk and can cover the entire contour of the aircraft as the key point to determine the region of interest of the aircraft;

[0017] Based on the region of interest, perform key point annotation on the original sample images and add sample labels.

[0018] Optionally, the specific operation of S2 is:

[0019] Perform grayscale processing and smoothing filtering on the labeled sample images to obtain optimized images;

[0020] Perform background suppression processing and binarization processing on the optimized images, and extract the aircraft target region from the binarized images to complete image preprocessing.

[0021] Optionally, the method further includes:

[0022] Perform left-right mirror flipping, up-down mirror flipping and rotation operations on each preprocessed sample image with a sample label to obtain augmented images;

[0023] Divide the dataset composed of the augmented images and the preprocessed sample images according to a preset ratio to obtain a training set and a validation set.

[0024] Optionally, in S3, the initial aircraft detection model is built based on YOLOv5 and includes: an input layer, a backbone network, a neck network and an output layer;

[0025] An input layer, which is used to receive the sample images input into the model and perform data augmentation;

[0026] The backbone network adopts a MobileNet network, which is used to extract image features;

[0027] The neck network adopts an FPN_PAN structure and embeds an attention mechanism. The attention mechanism includes a channel attention module and a spatial attention module, which are used to improve the ability to extract key image features and describe image features from the global receptive field;

[0028] The output layer uses SIoU as the loss function, which is used to correct the spatial direction factor between the true bounding box of the training set and the predicted bounding box output by the model, and outputs the object detection box.

[0029] Optionally, the MobileNet network includes: a convolutional layer, several depthwise separable convolutional layers, a global average pooling layer, and a fully connected layer; among them, the depthwise separable convolutional layer includes a depth convolutional kernel and a point convolutional kernel;

[0030] The convolutional layer is used to perform edge compensation processing and convolutional operations on the input initial image, and then obtain a feature map through BN processing and the ReLU activation function;

[0031] The global average pooling layer is used to perform average pooling on the feature map output by the last depthwise separable convolutional layer, so that the size of the final output feature map is 1×1;

[0032] The number of neurons in the fully connected layer is the number of image key points, and classification is performed through a Softmax classifier.

[0033] Optionally, the specific operation of S4 is as follows:

[0034] Initialize the model parameters, and input the training set into the initial aircraft detection model in batches for model training;

[0035] During the model training process, the validation set is used to adjust the parameters of the model through the mini-batch stochastic gradient descent method and the Adam optimizer. When the minimum loss error of the validation set reaches the minimum value, the model training is completed.

[0036] Optionally, the specific operation of S6 is as follows:

[0037] Add the first frame image with a detection box in the video to be recognized to an empty trajectory chain, and sequentially traverse all the images in the video to be recognized;

[0038] Predict the prediction box of the target aircraft in the next frame of image based on the last frame of image in each trajectory chain, match it with the detection box in the currently traversed image, and determine the trajectory chain of the target aircraft as the movement trajectory of the target aircraft;

[0039] Obtain the overlap degree between the movement trajectory of the target aircraft and the movement trajectory of the own aircraft. If the overlap degree exceeds the preset threshold, immediately send an alarm message through the on-board warning system.

[0040] A real-time detection system for wingtip conflicts on the apron based on deep learning, applying the real-time detection method for wingtip conflicts on the apron based on deep learning as described in any one of the above, includes:

[0041] An acquisition module for collecting several original sample images of different types of aircraft;

[0042] A labeling module for labeling the key points of the original sample images and adding sample labels;

[0043] A preprocessing module for preprocessing the labeled sample images and dividing the preprocessed sample images into a training set and a validation set according to a preset ratio;

[0044] A model construction module for constructing an improved YOLO network architecture based on the YOLO algorithm to obtain an initial aircraft detection model;

[0045] A model training module for inputting the training set and the validation set into the initial aircraft detection model for model training until the loss function converges to obtain a trained aircraft detection model;

[0046] A detection module for inputting the video to be recognized collected by a certain aircraft on the apron into the trained aircraft detection model to obtain the detection box of the target aircraft in each frame of the video to be recognized;

[0047] A judgment module for tracking the trajectory of the target aircraft in the detection box, judging whether the trajectory of the own aircraft coincides with the trajectory of the target aircraft. If there is a coincidence, immediately send an alarm message to the on-board warning system of the own aircraft.

[0048] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a real-time detection method and system for wingtip conflicts on the apron based on deep learning. An attention mechanism is added to the YOLO algorithm to construct an improved YOLO network architecture. The model is trained using the collected aircraft images of different models, which can extract more effective key features of the images, improve the detection accuracy and efficiency, more quickly and accurately judge whether there is a potential wingtip accident and remind the staff to take relevant measures in time, and improve the safety of aircraft activities on the apron. Description of the Drawings

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0050] Figure 1 It is a flowchart of the real-time detection method for apron wingtip conflicts provided by the present invention based on deep learning;

[0051] Figure 2 It is a structural diagram of the real-time detection system for apron wingtip conflicts provided by the present invention based on deep learning. Detailed implementation manners

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0053] The embodiment of the present invention discloses a real-time detection method for apron wingtip conflicts based on deep learning, as Figure 1 shown, including the following steps:

[0054] S1. Collect a number of original sample images of different types of aircraft, label the key points of the original sample images, and add sample labels;

[0055] S2. Preprocess the labeled sample images, and divide the preprocessed sample images into a training set and a validation set according to a preset ratio;

[0056] S3. Build an improved YOLO network architecture based on the YOLO algorithm to obtain an initial aircraft detection model;

[0057] S4. Input the training set and the validation set into the initial aircraft detection model for model training until the loss function converges to obtain a trained aircraft detection model;

[0058] S5. Input the video to be recognized collected by a certain aircraft on the apron into the trained aircraft detection model to obtain the detection frames of the target aircraft on each frame image of the video to be recognized;

[0059] S6. Track the trajectory of the target aircraft in the detection frame, judge whether the trajectory of the own aircraft coincides with that of the target aircraft. If there is a coincidence, immediately send an alarm message to the on-board warning system of the own aircraft.

[0060] Further, in this embodiment, the specific operation of S1 is as follows:

[0061] Take pictures of different models of aircraft from different shooting angles and distances to obtain a number of original sample images;

[0062] Based on the differences in aircraft models, select the position in the aircraft structure that is most prone to collision risk and can cover the entire outline of the aircraft as the key point to determine the region of interest of the aircraft;

[0063] Based on the region of interest, perform key point annotation on the original sample images and add sample labels.

[0064] Further, in this embodiment, the specific operation of S2 is as follows:

[0065] Perform grayscale processing and smoothing filtering on the labeled sample images to obtain an optimized image;

[0066] Perform background suppression processing and binarization on the optimized image, and extract the aircraft target region from the binarized image to complete image preprocessing.

[0067] Specifically, converting the labeled sample images into grayscale images and performing smoothing filtering can reduce the interference of noise in the images and facilitate subsequent processing; performing background suppression processing on the optimized images and performing binarization through an adaptive threshold method, detecting the connected components of the binarized images and extracting the area and centroid information of the connected components, and taking the connected component with the area greater than a given value and the smallest centroid coordinates as the aircraft target region to obtain the preprocessed sample images.

[0068] Further, in another implementation, the method further includes:

[0069] Perform left-right mirror flipping, up-down mirror flipping, and rotation operations on each preprocessed sample image with a sample label to obtain augmented images;

[0070] Divide the dataset composed of the augmented images and the preprocessed sample images according to a preset ratio to obtain a training set and a validation set.

[0071] Specifically, performing a rotation operation on the sample images, that is, taking the center of the sample image as the origin, and rotating both the label of the sample image and the sample image itself by a certain angle relative to the origin. In this embodiment, by augmenting the sample images, more reference data can be provided for subsequent model training, enhancing the model's detection ability for aircraft.

[0072] Further, in this embodiment, in S3, the initial aircraft detection model is built based on YOLOv5 and includes: an input layer, a backbone network, a neck network, and an output layer;

[0073] The input layer is used to receive the sample images input into the model and perform data augmentation;

[0074] The backbone network adopts the MobileNet network to extract image features;

[0075] The neck network adopts the FPN_PAN structure and embeds an attention mechanism. The attention mechanism includes a channel attention module and a spatial attention module, which are used to improve the ability to extract key image features and describe image features from the global receptive field;

[0076] The output layer uses SIoU as the loss function to correct the spatial direction factor between the true bounding box of the training set and the predicted bounding box output by the model, and outputs the target detection box.

[0077] Specifically, introducing an attention mechanism based on the YOLOv5 architecture can improve the network's ability to extract features of complex information, that is, it can improve the ability to extract key features of aircraft images, enabling the model to more clearly recognize the differences between different structures, which is helpful for the detection of target aircraft. Among them, the neck network is constructed based on the FPN_PAN structure. The horizontal axis of the FPN feature pyramid structure is regarded as the scale axis to extract scale-invariant feature variables, and the pyramid feature map is evenly adjusted to a high-resolution feature pyramid map and connected to the extracted scale-invariant feature variables for the feature region of aircraft detection by the YOLO model.

[0078] The calculation formula of SIoU is:

[0079]

[0080] In the formula: λ represents the distance between the center point of the predicted bounding box and the center point of the true bounding box, and the angle formed by the connection line and the vertical line of the two points' heights; ζ represents the similarity degree of the shapes of the predicted bounding box and the true bounding box; IoU represents the IoU value between the predicted bounding box and the true bounding box.

[0081] Furthermore, the MobileNet network includes: a convolutional layer, several depthwise separable convolutional layers, a global average pooling layer, and a fully connected layer; among them, the depthwise separable convolutional layer includes a depth convolution kernel and a point convolution kernel;

[0082] The convolutional layer is used to perform edge compensation processing and convolutional operations on the input initial image, and then obtain the feature map through BN processing and the ReLU activation function;

[0083] The global average pooling layer is used to perform average pooling on the feature map output by the last depthwise separable convolutional layer, so that the size of the final output feature map is 1×1;

[0084] The number of neurons in the fully connected layer is the number of image key points, and classification is performed by a Softmax classifier.

[0085] Further, in this embodiment, the specific operation of S4 is as follows:

[0086] Initialize the model parameters, and input the training set into the initial aircraft detection model in batches for model training;

[0087] During the model training process, the validation set is used to adjust the model parameters by the mini-batch stochastic gradient descent method and the Adam optimizer. When the minimum loss error of the validation set reaches the minimum value, the model training is completed.

[0088] Specifically, when training the model with the training set and the validation set, the model parameters that minimize the loss error of the validation set will be saved as the optimal parameters of the aircraft detection model. When the continuously recorded minimum loss error of the validation set starts to become larger than the previously recorded minimum value, the model stops training. Through this method, this embodiment can obtain an aircraft detection model that better meets the requirements, improve the accuracy of model prediction, and timely detect target aircraft that may pose a collision risk to avoid greater losses.

[0089] Further, in this embodiment, the specific operation of S6 is as follows:

[0090] Add the first frame picture with a detection box in the video to be recognized to an empty track chain, and sequentially traverse all pictures in the video to be recognized;

[0091] According to the last frame picture in each track chain, predict the prediction box of the target aircraft on the next frame picture and match it with the detection box on the currently traversed picture, and determine the track chain of the target aircraft as the movement track of the target aircraft;

[0092] Obtain the overlap degree between the movement track of the target aircraft and the movement track of the own aircraft. If the overlap degree exceeds the preset threshold, an alarm message will be immediately sent through the on-board warning system.

[0093] Specifically, when predicting the prediction box of the target aircraft in the next frame of image based on the last frame of image, the Kalman filter algorithm can be used; then, when matching the prediction box in the next frame of image with the detection box in the currently traversed image, it is determined whether the number of images continuously with labels in the trajectory chain exceeds a preset number. If it does not exceed the preset number, IOU matching is performed, which can comprehensively consider the overlap degree, center point distance, and aspect ratio consistency between the predicted bounding box and the true bounding box, improving the accuracy of target aircraft positioning; if it exceeds the preset number, the prediction box is cascaded and matched with each traversed detection box, and the motion trajectory of the target aircraft is obtained according to the completed trajectory chain.

[0094] And Figure 1 Corresponding to the method described above, an embodiment of the present invention further provides a real-time detection system for apron wingtip conflicts based on deep learning, which is used for Figure 1 For the specific implementation of the method in, a real-time detection system for apron wingtip conflicts based on deep learning provided by an embodiment of the present invention can be applied to computer terminals or various mobile devices, such as Figure 2 As shown, specifically including:

[0095] An acquisition module, which is used to collect several original sample images of different models of aircraft;

[0096] A labeling module, which is used to label the key points of the original sample images and add sample labels;

[0097] A preprocessing module, which is used to preprocess the labeled sample images and divide the preprocessed sample images into a training set and a validation set according to a preset ratio;

[0098] A model construction module, which is used to construct an improved YOLO network architecture based on the YOLO algorithm to obtain an initial aircraft detection model;

[0099] A model training module, which is used to input the training set and the validation set into the initial aircraft detection model for model training until the loss function converges, obtaining a trained aircraft detection model;

[0100] A detection module, which is used to input the to-be-recognized video collected by a certain aircraft on the apron into the trained aircraft detection model to obtain the detection box of the target aircraft in each frame of the to-be-recognized video;

[0101] A judgment module, which is used to track the trajectory of the target aircraft in the detection box, judge whether the trajectory of this aircraft coincides with that of the target aircraft, and if there is a coincidence, immediately send an alarm message to the on-board warning system of this aircraft.

[0102] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For related parts, reference can be made to the description in the method section.

[0103] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A real-time detection method for apron wingtip conflicts based on deep learning, characterized in that, It includes the following steps: S1. Collect several original sample images of different models of aircraft, label the key points of the original sample images, and add sample labels; S2. Preprocess the labeled sample images and divide the preprocessed sample images into a training set and a validation set according to a preset ratio; S3. Build an improved YOLO network architecture based on the YOLO algorithm to obtain an initial aircraft detection model; The initial aircraft detection model is built based on YOLOv5 and includes: an input layer, a backbone network, a neck network, and an output layer; the input layer is used to receive the sample images input into the model and perform data augmentation; the backbone network uses the MobileNet network to extract image features; the neck network uses the FPN_PAN structure and embeds an attention mechanism, and the attention mechanism includes a channel attention module and a spatial attention module, which are used to improve the extraction ability of key image features and describe image features from the global receptive field; the output layer uses SIoU as the loss function to correct the spatial direction factor between the true bounding box of the training set and the predicted bounding box output by the model, and outputs the object detection box; The MobileNet network includes: a convolutional layer, several depthwise separable convolutional layers, a global average pooling layer, and a fully connected layer; among them, the depthwise separable convolutional layer includes a depth convolution kernel and a point convolution kernel; the convolutional layer is used to perform edge compensation processing and convolution operation on the input initial image, and then obtain a feature map through BN processing and ReLU activation function; the global average pooling layer is used to perform average pooling on the feature map output by the last depthwise separable convolutional layer, so that the size of the final output feature map is 1×1; the number of neurons in the fully connected layer is the number of key points of the image, and classification is performed through a Softmax classifier; S4. Input the training set and the validation set into the initial aircraft detection model for model training until the loss function converges to obtain a trained aircraft detection model; S5. Input the video to be recognized of an aircraft on the apron into the trained aircraft detection model to obtain the detection box of the target aircraft on each frame image of the video to be recognized; S6. Track the trajectory of the target aircraft in the detection box, judge whether the trajectory of this aircraft coincides with the trajectory of the target aircraft, and if there is a coincidence, immediately send an alarm message to the on-board warning system of this aircraft.

2. The real-time detection method for apron wingtip conflicts based on deep learning according to claim 1, characterized in that, The specific operation of S1 is: Take pictures of different models of aircraft at different shooting angles and shooting distances to obtain several original sample images; Based on the different aircraft models, select the position in the aircraft structure that is most prone to collision risk and can cover the entire contour of the aircraft as the key point to determine the region of interest of the aircraft; Based on the region of interest, label the key points of the original sample images and add sample labels.

3. A real-time detection method for apron wingtip conflicts based on deep learning according to claim 1, characterized in that, The specific operation of S2 is: Perform grayscale processing and smoothing filtering on the labeled sample images to obtain an optimized image; Perform background suppression processing and binarization processing on the optimized image, and extract the aircraft target region from the binarized image to complete image preprocessing.

4. A real-time detection method for apron wingtip conflicts based on deep learning according to claim 1, characterized in that It also includes: Perform left-right mirror flipping, up-down mirror flipping, and rotation operations on each preprocessed sample image with a sample label to obtain augmented images; Divide the dataset composed of the augmented images and the preprocessed sample images according to a preset ratio to obtain a training set and a validation set.

5. A real-time detection method for apron wingtip conflicts based on deep learning according to claim 1, characterized in that The specific operation of S4 is as follows: Initialize the model parameters, and input the training set into the initial aircraft detection model in batches for model training; During the model training process, use the validation set to adjust the model parameters through the mini-batch stochastic gradient descent method and the Adam optimizer. When the minimum loss error of the validation set reaches the minimum value, the model training is completed.

6. The real-time detection method for apron wingtip conflicts based on deep learning according to claim 1, wherein The specific operation of S6 is as follows: Add the first frame image with a detection box in the video to be recognized to an empty trajectory chain, and sequentially traverse all the images in the video to be recognized; Based on the last frame image in each trajectory chain, predict the prediction box of the target aircraft on the next frame image and match it with the detection box on the currently traversed image, and determine the trajectory chain of the target aircraft as the movement trajectory of the target aircraft; Obtain the coincidence degree between the movement trajectory of the target aircraft and the movement trajectory of the own aircraft. If the coincidence degree exceeds the preset threshold, immediately send an alarm message through the on-board warning system.

7. A real-time detection system for apron wingtip conflicts based on deep learning, which applies the real-time detection method for apron wingtip conflicts based on deep learning according to any one of claims 1-6, characterized in that Including: An acquisition module for collecting a number of original sample images of different models of aircraft; A labeling module for labeling the key points of the original sample images and adding sample labels; A preprocessing module for preprocessing the labeled sample images and dividing the preprocessed sample images into a training set and a validation set according to a preset ratio; A model construction module for constructing an improved YOLO network architecture based on the YOLO algorithm to obtain an initial aircraft detection model; A model training module for inputting the training set and the validation set into the initial aircraft detection model for model training until the loss function converges to obtain a trained aircraft detection model; A detection module for inputting the video to be recognized collected by a certain aircraft on the apron into the trained aircraft detection model to obtain the detection box of the target aircraft on each frame image in the video to be recognized; A judgment module for tracking the trajectory of the target aircraft in the detection box, judging whether the trajectory of the own aircraft coincides with the trajectory of the target aircraft. If there is a coincidence, immediately send an alarm message to the on-board warning system of the own aircraft.

Images