Automatic identification method of airport support nodes based on relative position learning

Through relative position learning and deep learning combined with YOLOV7 model, the key points of the aircraft are identified, which solves the problem of poor identification results in rainy and fog or night environments in the existing technology, and realizes automatic identification of airport guarantee nodes.

CN118799813BActive Publication Date: 2024-12-03CIVIL AVIATION CHENGDU ELECTRONIC TECH CO LTD +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411261329.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-12-03
Estimated Expiration
2044-09-10

AI Technical Summary

Technical Problem

The existing automatic entry method of airport guarantee nodes based on image recognition is poor in rainy and fog or night environments, resulting in the system being unable to effectively detect guarantee nodes.

Method used

Using the relative position learning method, first identify the position of the aircraft, and then predict the relative position of the guaranteed vehicle and the aircraft based on the position posture of the aircraft. Deep learning and improved YOLOV7 model are used to detect the image data, and identify the position data of the key points of the aircraft. Perform automatic identification.

Benefits of technology

In rainy and fog or night environments, it can identify key parts of the aircraft with high accuracy, solve the problem of poor image recognition effect, and realize automatic identification of airport support nodes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799813B_ABST
    Figure CN118799813B_ABST
Patent Text Reader

Abstract

The present invention discloses an automatic identification method for airport support nodes based on relative position learning, which relates to the field of apron support technology, including: S1, according to the relative position relationship between the support vehicle and the aircraft, a data set is produced to mark the relative position data of the support vehicle and the aircraft; S2, the data set is trained by deep learning to obtain a prediction model; S3, the image data transmitted in real time from the airport is detected by an improved YOLOV7 model to obtain the key point position data of the aircraft; S4, the key point position data is input into the prediction model to complete the automatic identification of the airport support nodes. The present invention makes up for the shortcomings of the support node identification method that completely relies on image detection, and improves the detection efficiency of the support node.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of apron support, and in particular to an automatic identification method of airport support nodes based on relative position learning. Background Art

[0002] The airport security node system is a vital part of modern airport operations. It covers a series of technologies and facilities to ensure the efficiency, safety and reliability of various airport operations. In traditional airport operation management, the collection of security node information usually relies on manual observation and manual reporting, which has some problems. Relying solely on manual observation and manual reporting will face the following problems: increasing the workload of personnel, which may affect the quality of work; the real-time performance of data is poor, which may lead to untimely information; manual reporting is prone to omissions, false reports and other problems, which affects the accuracy and reference of the data. The airport security node system can solve the above problems, and through the integration of multiple subsystems and equipment, real-time detection and feedback coordination can be carried out to ensure the order of the airport.

[0003] At present, image processing technology plays a very important role in flight support. Through intelligent sensing technology, sensor networks and surveillance cameras are used to monitor the situation in various areas of the airport in real time. The collected image data can be used to detect the location and movement of aircraft, vehicles, luggage and personnel through image recognition technology, as well as environmental factors such as weather, temperature and visibility. Establish a reliable communication and network infrastructure to ensure information exchange and interconnection between various devices. This includes technologies such as local area networks, wireless networks, and satellite communications to ensure the stable operation of the system in complex environments. Integrate with the systems of air traffic control departments and airlines to ensure flight safety and normal operation.

[0004] The existing automatic entry of support nodes based on image recognition requires identifying the type of support vehicle based on the video and being able to track the vehicle during the entire video process. However, this technology may have serious errors when the support vehicle is obscured or exposed to rain, fog and light, which greatly reduces the recognition effect of the entire system. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the airport support node automatic identification method based on relative position learning provided by the present invention does not directly identify the support vehicle, but first identifies the position of the aircraft, and predicts the relative position of the support vehicle and the aircraft based on the aircraft position. As long as a vehicle enters the identification area, it can be considered as the appearance of the corresponding support vehicle. Since the method only needs to identify several key parts of the aircraft based on image recognition technology, such as the nose and tail, these two key parts have high recognizability and can be well identified in rain, fog or at night, and there is almost no problem of being blocked. It solves the problem that the existing support node automatic entry method based on image recognition cannot perform support node detection when the image recognition effect is poor.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: an automatic identification method of airport support nodes based on relative position learning, comprising:

[0007] S1. Create a data set with the relative position data of the support vehicle and the aircraft according to the relative position relationship between the support vehicle and the aircraft;

[0008] S2. Use deep learning to train the data set to obtain a prediction model;

[0009] S3. Use the improved YOLOV7 model to detect the real-time image data transmitted from the airport to obtain the key point position data of the aircraft;

[0010] S4. Input the key point location data into the prediction model to complete the automatic identification of airport support nodes.

[0011] Further: S1 includes:

[0012] S11, randomly transforming the 3D image of the aircraft at different shooting angles and shooting distances, and marking key points of the data to obtain aircraft posture image data;

[0013] S12. Obtain relative position features based on the aircraft posture image data and the corresponding support vehicle position, and use the relative position features as a data set for annotating the relative position data of the support vehicle and the aircraft.

[0014] Further: in S12, the relative position features include the nose x-coordinate, the nose y-coordinate, the tail x-coordinate, the tail y-coordinate, the aircraft orientation, the upper left corner coordinate of the support vehicle detection frame, and the lower right corner coordinate of the support vehicle detection frame.

[0015] Further: in S2, the prediction model includes an input layer, a first hidden layer, a second hidden layer and an output layer connected in sequence;

[0016] The first hidden layer includes 64 neurons, and the second hidden layer includes 32 neurons.

[0017] Further: In S3, the method for improving the YOLOV7 model to process the image data transmitted in real time from the airport includes:

[0018] S31. Input the real-time image data from the airport into the improved YOLOV7 model, and extract feature maps of three different sizes by improving the backbone network and neck network of the YOLOV7 model;

[0019] S32. Use the decoupling head to predict the feature maps of three different sizes respectively, and obtain the Cls feature for classification, the Reg feature for positioning, and the Obj feature for the confidence task;

[0020] S33, the Cls feature used for classification, the Reg feature used for positioning, and the Obj feature used for the confidence task are combined into a final feature map through a stacking operation, and the final feature map is used as the key point position data of the aircraft.

[0021] Further: in said S32, the decoupling head includes a classification branch and a regression branch;

[0022] The classification branch includes two 3×3 convolutional layers and one 1×1 convolutional layer connected in sequence;

[0023] The classification branch outputs a Cls feature for classification;

[0024] The regression branch includes two 3×3 convolutional layers connected in sequence and a 1×1 convolutional layer in parallel;

[0025] The regression branch outputs Reg features for positioning and Obj features for confidence tasks.

[0026] Further: S4 includes:

[0027] S41, inputting the key point position data into the prediction model, and outputting the detection frame of the security vehicle through the prediction model;

[0028] S42, tracking the movement trajectory of the support vehicle segment in the detection frame of the support vehicle, obtaining the start time node and the end time node of the airport support according to the movement trajectory of the support vehicle segment, and completing the automatic identification of the airport support node.

[0029] The beneficial effects of the present invention are:

[0030] The airport support node automatic identification method based on relative position learning provided by the present invention does not directly identify the support vehicle, but first identifies the posture of the aircraft, and predicts the relative position of the support vehicle and the aircraft based on the posture of the aircraft. As long as a vehicle enters the identification area, it can be considered that the corresponding support vehicle appears.

[0031] Since this method only needs to identify several key parts of the aircraft based on image recognition technology, such as the nose and the tail, these two key parts have high identifiability and can be well identified in rain, fog or at night, and there is almost no problem of being blocked. It solves the problem that the existing automatic entry method of support nodes based on image recognition cannot perform support node detection when the image recognition effect is poor. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of the automatic identification method of airport support nodes based on relative position learning.

[0033] Figure 2 Schematic diagram of key points of the 3D model of the aircraft at different distances.

[0034] Figure 3 Schematic diagram of key point annotation of the aircraft 3D model from different top-down angles.

[0035] Figure 4 A schematic diagram showing the front and back sides of the aircraft relative positions.

[0036] Figure 5 Schematic diagram of the prediction model structure.

[0037] Figure 6 Schematic diagram of the automatic identification method of airport support nodes based on relative position learning. DETAILED DESCRIPTION

[0038] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0039] like Figure 1 As shown, in one embodiment of the present invention, a method for automatically identifying airport support nodes based on relative position learning is provided, comprising:

[0040] S1. Create a data set with the relative position data of the support vehicle and the aircraft according to the relative position relationship between the support vehicle and the aircraft;

[0041] S2. Use deep learning to train the data set to obtain a prediction model;

[0042] S3. Use the improved YOLOV7 model to detect the real-time image data transmitted from the airport to obtain the key point position data of the aircraft;

[0043] S4. Input the key point location data into the prediction model to complete the automatic identification of airport support nodes.

[0044] The S1 includes:

[0045] S11, randomly transforming the 3D image of the aircraft at different shooting angles and shooting distances, and marking key points of the data to obtain aircraft posture image data;

[0046] S12. Obtain relative position features based on the aircraft posture image data and the corresponding support vehicle position, and use the relative position features as a data set for annotating the relative position data of the support vehicle and the aircraft.

[0047] In one embodiment of the present invention, Figure 2 and Figure 3 As shown, in the S11, Figure 2 The diagram shows the key points of the 3D model of the aircraft at different distances. Figure 2 (a) indicates that the shooting distance is close, Figure 2 (b) indicates that the shooting distance is medium distance. Figure 2 (c) indicates that the shooting distance is long;

[0048] Figure 3 It shows screenshots of the 3D model at three top-down angles: 0°, 30°, and 60°. Figure 3 (a) represents the top view angle of 0°, Figure 3 (b) represents a top-down angle of 30°. Figure 3 (c) represents a top-down angle of 60°;

[0049] Corresponding annotation data is generated for each different combination of angle and distance, marking the position of the aircraft nose and tail as well as the relative positions of different support vehicles in the current aircraft posture. These annotated position data are used as training data for support vehicle position prediction.

[0050] In one embodiment of the present invention, the relative position feature selection process in S12 is a delicate and critical task, which includes the screening of the relative positions of the aircraft and the support vehicles, and usually involves different types of relative position features such as the nose position, the tail position, and the passenger elevator position. When performing feature selection, it is first necessary to observe and analyze the image or video to determine the position and type of each feature. Then, according to the shape, size, position and other characteristics of the target, different calculation formulas are used to determine its specific annotation position and attributes.

[0051] In this embodiment, the selected relative position features include the nose x-coordinate, nose y-coordinate, tail x-coordinate, tail y-coordinate, aircraft orientation, upper left corner coordinate of the support vehicle detection frame, and lower right corner coordinate of the support vehicle detection frame;

[0052] Since the origin of the image is in the upper left corner of the image, we use the distance from the center point of the nose annotation box to the x coordinate of the upper left corner of the image as the x coordinate of the nose, the distance from the center point of the nose annotation box to the y coordinate of the upper left corner of the image as the y coordinate of the nose, the distance from the center point of the tail annotation box to the x coordinate of the upper left corner of the image as the x coordinate of the tail, and the distance from the center point of the tail annotation box to the y coordinate of the upper left corner of the image as the y coordinate of the tail;

[0053] Therefore, the x-coordinate of the nose is the x-coordinate of the center point of the nose annotation box; the y-coordinate of the nose is the y-coordinate of the center point of the nose annotation box; the x-coordinate of the tail is the x-coordinate of the center point of the tail annotation box, and the y-coordinate of the tail is the y-coordinate of the center point of the tail annotation box;

[0054] like Figure 4 As shown, in the feature selection of the aircraft orientation, Figure 4 (a) The nose of the aircraft facing the nose of the video is marked as the positive nose, and the nose of the aircraft facing the tail of the video is marked as the positive tail. Figure 4 (b) The nose of the aircraft with the tail facing the video is marked as the reverse nose, and the tail of the aircraft with the tail facing the video is marked as the reverse tail, which improves the recognition accuracy of passenger elevators.

[0055] Since the relative position of the support vehicle and the aircraft needs to be predicted as the detection frame position of the support vehicle, and the detection frame only needs to determine the coordinates of the upper left corner and the lower right corner of the detection frame, it is worth pointing out that it is not easy to predict the absolute position of the support vehicle in the picture, but it is easier to predict the relative position of the support vehicle and the aircraft. Therefore, the coordinates of the upper left corner and the lower right corner of the detection frame are the relative positions of the detection frame and the nose of the aircraft;

[0056] The detection of the security vehicle is done by determining the x and y coordinates of the security vehicle. The detection frame is determined by the x, y coordinates and the width and height of the security vehicle. The center position of the security vehicle is: 保障车辆 =X 保障车辆x坐标 -X 机头;

[0057] Where X 保障车辆 is the x coordinate of the vehicle relative to the nose, X 保障车辆x坐标 is the absolute x coordinate of the security vehicle, X 机头 is the actual x-coordinate of the nose;

[0058] Ensure that the coordinates of the upper left corner of the vehicle detection frame = (X 保障车辆 -W 保障车辆 / 2, y - H 保障车辆 / 2), to ensure that the coordinates of the lower right corner of the vehicle detection frame = (X 保障车辆 + W 保障车辆 / 2,y+ H 保障车辆 / 2), where W 保障车辆 is the width of the vehicle detection frame, H 保障车辆 It is the height of the security vehicle detection frame, so that the coordinates of the upper left corner and the lower right corner can be obtained, thereby determining the position of the entire security vehicle detection frame.

[0059] pass Figure 4 (a) and Figure 4 (b) The relative position training data we obtained is shown in Table 1:

[0060] Table 1 Relative position data of aircraft and its support vehicles based on 3D model

[0061] Serial number Head x coordinate Head y coordinate Tail x coordinate Tail y coordinate Aircraft direction Ensure the coordinates of the upper left corner of the vehicle detection frame Ensure the coordinates of the lower right corner of the vehicle detection frame Protected vehicle type Figure 4 (a) 0.493229 0.492969 0.497396 0.538281 Positive 0.614323 0.561719 Passenger elevator Figure 4 (b) 0.271094 0.508984 0.599740 0.513672 Reverse 0.380208 0.564844 Passenger lift

[0062] like Figure 5 As shown, in S2, the prediction model includes an input layer, a first hidden layer, a second hidden layer and an output layer connected in sequence; the first hidden layer includes 64 neurons, and the second hidden layer includes 32 neurons.

[0063] Using the prepared feature annotation dataset, the preprocessed data is input into the prediction model for training, and the weights are updated through the back propagation algorithm to minimize the loss function. The model updates the weights through the back propagation algorithm to minimize the loss function.

[0064] like Figure 6As shown in the figure, the upper part describes how to identify the key point data of the aircraft based on the YOLOv7 model. At the input end, the batched image data is processed after feature processing and a series of data enhancement techniques. The backbone network is the core part of the YOLOv7 model, responsible for feature extraction and representation of the input image. It usually consists of a series of convolutional layers, pooling layers, and other basic operations to convert the input image into a feature map with higher-level semantic information. The main function of the backbone network is to extract features from the original image so that the model can understand the content and structure in the image.

[0065] The neck network is located behind the backbone network and is responsible for further processing and fusing information from feature maps at different levels to improve the model's detection performance of the target. The design of the neck network uses operations such as feature fusion, scale transformation, and attention mechanism to enhance the model's perception and target modeling capabilities.

[0066] Since the field of target detection includes two tasks, classification and prediction, the classification task mainly focuses on the similarity between the features extracted by the feature layer and the real category features, and the prediction task mainly focuses on the position coordinates of the prediction box and the real box to correct the parameters of the prediction bounding box. Since the goal is to collect the data in Table 1, in order to make the data in Table 1 as accurate as possible, it is necessary to make the parameter correction of the prediction bounding box as accurate as possible. The YOLO series of algorithms have always used the same feature map for classification and positioning tasks, and different tasks use the same shared parameters, so that the two tasks cannot focus on their respective targets, and their detection effect will deteriorate. In order to separate the classification and prediction tasks, the present invention replaces the prediction head at the network output end in YOLOv7 with a decoupled head structure.

[0067] In S3, the method for improving the YOLOV7 model to process the image data transmitted in real time from the airport includes:

[0068] S31. Input the real-time image data from the airport into the improved YOLOV7 model, and extract feature maps of three different sizes by improving the backbone network and neck network of the YOLOV7 model;

[0069] S32. Use the decoupling head to predict the feature maps of three different sizes respectively, and obtain the Cls feature for classification, the Reg feature for positioning, and the Obj feature for the confidence task;

[0070] S33, the Cls feature used for classification, the Reg feature used for positioning, and the Obj feature used for the confidence task are combined into a final feature map through a stacking operation, and the final feature map is used as the key point position data of the aircraft.

[0071] In the S32, the decoupling head includes a classification branch and a regression branch;

[0072] The classification branch includes two 3×3 convolutional layers and one 1×1 convolutional layer connected in sequence;

[0073] The classification branch outputs a Cls feature for classification;

[0074] The regression branch includes two 3×3 convolutional layers connected in sequence and a 1×1 convolutional layer in parallel;

[0075] The regression branch outputs Reg features for positioning and Obj features for confidence tasks.

[0076] The S4 includes:

[0077] S41, inputting the key point position data into the prediction model, and outputting the detection frame of the security vehicle through the prediction model;

[0078] S42, tracking the movement trajectory of the support vehicle segment in the detection frame of the support vehicle, obtaining the start time node and the end time node of the airport support according to the movement trajectory of the support vehicle segment, and completing the automatic identification of the airport support node.

[0079] Once a vehicle is detected at the predicted relative position, it means that the corresponding support vehicle has entered the work area, and its movement trajectory can be tracked. This can be achieved by matching the bounding box of the vehicle between consecutive frames. When a vehicle is detected, the timestamp of its appearance is recorded. At the same time, when the vehicle disappears from the image or video, the corresponding timestamp is also recorded. By analyzing these recorded timestamp data, the time when the vehicle appears and disappears at the predicted position can be obtained. The appearance time of each detected vehicle, that is, after various vehicles are in place, the start time of the support node can be obtained, and it is executed according to the normal procedure. After the start of the support node, each vehicle enters in an orderly manner. After boarding is completed, each vehicle evacuates in an orderly manner. After the disappearance time of each vehicle is obtained, the end time of the support node can be obtained when the aircraft takes off.

[0080] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An automatic identification method for airport support nodes based on relative position learning, characterized in that: include: S1. Create a data set with the relative position data of the support vehicle and the aircraft according to the relative position relationship between the support vehicle and the aircraft; S2. Use deep learning to train the data set to obtain a prediction model; S3. Use the improved YOLOV7 model to detect the real-time image data transmitted from the airport to obtain the key point position data of the aircraft; The methods for improving the YOLOV7 model to process the real-time image data transmitted from the airport include: S31. Input the real-time image data from the airport into the improved YOLOV7 model, and extract feature maps of three different sizes by improving the backbone network and neck network of the YOLOV7 model; S32. Use the decoupling head to predict the feature maps of three different sizes respectively, and obtain the Cls feature for classification, the Reg feature for positioning, and the Obj feature for the confidence task; S33, combining the Cls feature for classification, the Reg feature for positioning, and the Obj feature for the confidence task into a final feature map through a stacking operation, and using the final feature map as the key point position data of the aircraft; S4. Input the key point location data into the prediction model to complete the automatic identification of airport support nodes.

2. The method for automatically identifying airport support nodes based on relative position learning according to claim 1 is characterized in that: The S1 includes: S11, randomly transforming the 3D image of the aircraft at different shooting angles and shooting distances, and marking key points of the data to obtain aircraft posture image data; S12. Obtain relative position features based on the aircraft posture image data and the corresponding support vehicle position, and use the relative position features as a data set for annotating the relative position data of the support vehicle and the aircraft.

3. The method for automatically identifying airport support nodes based on relative position learning according to claim 2 is characterized in that: In S12, the relative position feature includes the head x Coordinates, head y Coordinates, tail x Coordinates, tail y Coordinates, aircraft orientation, coordinates of the upper left corner of the support vehicle detection frame, and coordinates of the lower right corner of the support vehicle detection frame.

4. The method for automatically identifying airport support nodes based on relative position learning according to claim 1 is characterized in that: In S2, the prediction model includes an input layer, a first hidden layer, a second hidden layer and an output layer connected in sequence; The first hidden layer includes 64 neurons, and the second hidden layer includes 32 neurons.

5. The method for automatically identifying airport support nodes based on relative position learning according to claim 1 is characterized in that: In the S32, the decoupling head includes a classification branch and a regression branch; The classification branch includes two 3×3 convolutional layers and one 1×1 convolutional layer connected in sequence; The classification branch outputs a Cls feature for classification; The regression branch includes two 3×3 convolutional layers connected in sequence and a 1×1 convolutional layer in parallel; The regression branch outputs Reg features for positioning and Obj features for confidence tasks.

6. The method for automatically identifying airport support nodes based on relative position learning according to claim 1 is characterized in that: The S4 includes: S41, inputting the key point position data into the prediction model, and outputting the detection frame of the security vehicle through the prediction model; S42, tracking the movement trajectory of the support vehicle segment in the detection frame of the support vehicle, obtaining the start time node and the end time node of the airport support according to the movement trajectory of the support vehicle segment, and completing the automatic identification of the airport support node.

Citation Information

Patent Citations

  • Method for calculating flight guaranteed node time based on vehicle positions

    CN106251710A

  • Automatic identification method and system for key time nodes of airport scene

    CN115797822A

  • YOLOv6-based bridge disease inspection method and system

    CN117151680A

  • Ground based security surveillance system for aircraft and other commercial vehicles

    US6545601B1

  • Airport flight zone real-time target detection method based on multiscale feature decoupling

    WO2024152477A1