A method for door tracking for automatic docking of a boarding bridge to a door

Abstract

The application discloses a cabin door tracking method for automatic docking of a boarding bridge with a cabin door, relates to the field of image recognition, and particularly relates to a recognition and tracking method for an aircraft cabin door. A camera at the head of the boarding bridge is used to collect a fuselage video in real time, a trained cabin door detection model is used to detect the cabin door in real time, and a tracker based on SIoU matching is combined with detection information of front and rear frames of the video to realize cabin door tracking. In addition to considering the overlapping degree between two detection boxes in matching, SIoU also considers the distance between the centers of the detection boxes and the angle of the connecting line of the centers, so that the accuracy of detection box matching is improved, and the occurrence of cabin door false detection and missed detection is reduced.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and in particular to a method for identifying and tracking aircraft cabin doors. Background Technology

[0002] Airport boarding bridges are movable, lift-up passageways used in airports to connect waiting areas with aircraft. With the development of airports and the entire transportation system, airports are becoming increasingly larger, and passenger capacity is growing daily. Consequently, the utilization rate of boarding bridges has increased significantly. As the link between the airport and aircraft, boarding bridges play a crucial role in airport operations, and the quality of boarding bridge docking technology directly affects airport operational efficiency.

[0003] Currently, the docking and disembarking of boarding bridges are mainly done manually by bridge operators in the control room. This requires a high level of skill, combining visual observation with manual controls, and is a cumbersome process. Airports incur significant annual costs for both personnel and training. Furthermore, manual operation is prone to errors, potentially leading to repeated adjustments, collisions with the aircraft, or other obstacles, resulting in low docking efficiency and accuracy.

[0004] With the development of artificial intelligence and computer vision, deep learning-based object detection models can now achieve high accuracy and real-time detection speed in hatch detection. However, if only object detection technology is used to track hatches in video frames, the interference of complex backgrounds in the image can lead to the false detection of multiple hatches or the failure to detect a target hatch in a single frame, which can affect the correct docking of the hatches.

[0005] Target tracking algorithms can combine information from multiple frames before and after a video to identify the correct target, resulting in more accurate tracking. Currently, most target detection-based trackers use Interchange of Units (IoU) to calculate the similarity between detection boxes in consecutive frames. However, IoU only considers the overlap between detection boxes and does not take into account factors such as the distance and angle between the bounding box centers, leading to low matching accuracy. Summary of the Invention

[0006] This invention addresses the problems of slow target door recognition speed and low accuracy in the prior art by designing a door tracking method for automatic docking of boarding bridges.

[0007] This invention proposes a door tracking method for automatic docking doors of boarding bridges, which includes a training phase and a tracking phase:

[0008] During the training phase, a certain number of aircraft door images are collected as training samples, and the training set is augmented using certain geometric transformations and color space transformations. Finally, the training samples are labeled with YOLO format labels, and the improved YOLO door detection model is trained for door detection.

[0009] During the tracking phase, the camera mounted on the boarding bridge is activated to capture real-time video of the aircraft. The trained door detection model is used to detect the door position in each frame in real time. The tracker uses the tracking information from previous frames to match the door in the current frame with the tracking trajectory of the previous frame and adds the correct tracking target to the tracking trajectory.

[0010] The specific steps of the training phase are as follows:

[0011] (1) Use the camera mounted on the boarding bridge to collect images of various aircraft fuselages under different locations, weather and lighting conditions as a dataset, or collect images of aircraft fuselages from the Internet to expand the dataset.

[0012] (2) Geometric transformations to augment images, such as translation, cropping, flipping, rotation, and color transformations;

[0013] (3) Label the dataset with YOLO format labels. Each YOLO format label corresponds to a txt text file for each image. Each line of the txt text file corresponds to a hatch in the image. Each line includes the target category, the center coordinates of the target bounding box, and the scale parameter of the bounding box.

[0014] (4) The dataset is split into training and test sets in a ratio of 4:1. The training set is input into the improved YOLO object detection network for training. The test set is used to test the detection accuracy and inference speed of the model. When the accuracy and inference speed meet the requirements, the trained object detection model is used for hatch detection in the tracking stage.

[0015] The specific steps of the tracking phase are as follows:

[0016] (1) Activate the camera mounted at the head of the boarding bridge to continuously collect video data of the aircraft fuselage;

[0017] (2) Input the video into the trained hatch detection model, detect the hatch in each frame of the video in real time, and output the detection box of the hatch in the video to the tracker.

[0018] (3) The tracker combines the tracking trajectory obtained from the previous frame to track the detection box in the current frame;

[0019] (4) The boarding bridge adjusts the docking angle and speed according to the tracking trajectory.

[0020] In step (3) of the tracking phase, let the output of the hatch detection model in frame f be set D. f d j The j-th detection result in the set; the activated tracking trajectory is stored in set T.a In the middle, t i This represents the i-th detection result; the completed tracking trajectories are stored in set T. f The confidence threshold for target detection is σ. l The threshold for tracking and matching is σ. SIoU The confidence threshold for completed tracking is σ. h The shortest tracking step size is t. min ;

[0021] The specific steps of the tracking method are as follows:

[0022] (1) Read the hatch detection result D of the next frame. f ;

[0023] (2) Using the threshold σ l Eliminate targets with low confidence levels;

[0024] (3) Traverse set T a All active tracking trajectories t i Find the detection set D of the current frame. f In the middle, with the tracking trajectory t i The SioU detected in the previous frame. SioU represents the IoU based on position loss information, with the largest d. j As d best If the calculated SIoU result is greater than the threshold σ SIoU , then d j Add trajectory t i and from set D f Remove from the middle; if not, then determine t. i Does it satisfy the condition that at least one detection result has a confidence level higher than σ? h Step size greater than t min If the condition is met, then t will be... i Add to set T f Finally, t i From set T a Remove from;

[0025] (4) For other unmatched d j Then, each of these will be created as a new tracking trajectory and added to set T. a middle;

[0026] (5) If not all frames are matched, continue with steps (1)-(5);

[0027] (6) After matching all frames, iterate through set T. a All activated trajectories t i Determine t i Does it satisfy the condition that at least one detection result has a confidence level higher than σ? hStep size greater than t min If the condition is met, then t will be... i Add to set T f If satisfied, t i Add to set T f Finally, t i From set T a Removed from the middle.

[0028] Furthermore, in step (3) of the tracking method, the SIoU loss of the tracking method includes four parts: IoU, angle loss, distance loss, and shape loss, and the specific calculation method is as follows:

[0029] (3.1) IoU:

[0030] IoU is the intersection-union ratio between two bounding boxes A and B. The formula is as follows:

[0031]

[0032] In the formula, Area(A) and Area(B) represent the area of ​​bounding box A and the area of ​​bounding box B, respectively; when the two bounding boxes completely overlap, the IoU is the largest, which is 1, and when the two bounding boxes do not overlap, the IoU is the smallest, which is 0.

[0033] (3.2) Angle loss A:

[0034] The formula for calculating angle loss is:

[0035]

[0036] in,

[0037]

[0038] In the formula, c h σ represents the difference in the vertical positions of the centers of the two hatch boundary frames, σ is the straight-line distance between the centers of the two hatch boundary frames, and x represents the sine value of the small angle α. The angle loss Λ is minimal (0) when the center points are aligned on the x-axis or y-axis, and Λ is maximum (1) when the angle between the connecting line and both the x-axis and the y-axis is 45°.

[0039] (3.3) Distance loss Δ:

[0040] The formula for calculating distance loss Δ is:

[0041]

[0042] in,

[0043] γ=2-Λ

[0044] In the formula, c w This is the difference in the horizontal position of the center of the two hatch boundary frames. Indicates the coordinates of the center point of the hatch boundary frame A. The coordinates of the center point of the hatch boundary frame B are given, and Λ represents the angle loss.

[0045] (3.4) Shape loss Ω:

[0046] The formula for calculating the shape loss Ω is:

[0047]

[0048] in,

[0049]

[0050] In the formula, w A h A The width and height of the hatch boundary frame A are respectively w B h B Define the width and height of the hatch boundary frame A.

[0051] (3.5) Total loss SIoU:

[0052]

[0053] This invention proposes a door tracking method for automated docking of aircraft doors on boarding bridges. The method involves real-time acquisition of aircraft fuselage video via cameras at the bridgehead, real-time detection of the doors using a trained door detection model, and door tracking achieved by a tracker based on SIoU matching combined with detection information from preceding and following video frames. SIoU matching considers not only the overlap between two detection frames but also the distance between the centers of the detection frames and the angle of the line connecting them, which improves the accuracy of detection frame matching and reduces false positives and false negatives in door detection. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the angle loss calculation principle provided in an embodiment of the present invention;

[0055] Figure 2 This is a flowchart of the tracking method provided in an embodiment of the present invention;

[0056] Figure 3 The effect diagram provided by the embodiment of the present invention. Detailed Implementation

[0057] A door tracking method for automatic docking doors of boarding bridges, comprising a training phase and a tracking phase:

[0058] During the training phase, a certain number of aircraft door images are collected as training samples, and the training set is augmented using certain geometric transformations and color space transformations. Finally, the training samples are labeled with YOLO format labels, and the improved YOLO door detection model is trained for door detection.

[0059] During the tracking phase, the camera mounted on the boarding bridge is activated to capture real-time video of the aircraft. The trained door detection model is used to detect the door position in each frame in real time. The tracker uses the tracking information from previous frames to match the door in the current frame with the tracking trajectory of the previous frame and adds the correct tracking target to the tracking trajectory.

[0060] The specific steps of the training phase are as follows:

[0061] (1) Use the camera mounted on the boarding bridge to collect images of various aircraft fuselages under different locations, weather and lighting conditions as a dataset, or collect images of aircraft fuselages from the Internet to expand the dataset.

[0062] (2) Perform geometric transformations on the images, such as translation, cropping, flipping, rotation, and color transformation; such as brightness transformation, noise addition, contrast adjustment, saturation adjustment, etc., to expand the number of training samples through data augmentation and improve the accuracy and robustness of the hatch detection model.

[0063] (3) Label the dataset with YOLO format labels. Each YOLO format label corresponds to a txt text file for each image. Each line of the txt text file corresponds to a hatch in the image. Each line includes the target category, the center coordinates of the target bounding box, and the scale parameter of the bounding box.

[0064] (4) The dataset is split into training and test sets in a ratio of 4:1. The training set is input into the improved YOLO object detection network for training. The test set is used to test the detection accuracy and inference speed of the model. When the accuracy and inference speed meet the requirements, the trained object detection model is used for hatch detection in the tracking stage.

[0065] The specific steps of the tracking phase are as follows:

[0066] (1) Activate the camera mounted at the head of the boarding bridge to continuously collect video data of the aircraft fuselage;

[0067] (2) Input the video into the trained hatch detection model, detect the hatch in each frame of the video in real time, and output the detection box of the hatch in the video to the tracker.

[0068] (3) The tracker combines the tracking trajectory obtained from the previous frame to track the detection box in the current frame;

[0069] (4) The boarding bridge adjusts the docking angle and speed according to the tracking trajectory.

[0070] In step (3) of the tracking phase, the specific process of the tracking method is as follows: Figure 2 As shown. Let the output of the hatch detection model in frame f be set D. f d j The j-th detection result in the set; the activated tracking trajectory is stored in set T. a In the middle, t i This represents the i-th detection result; the completed tracking trajectories are stored in set T. f The confidence threshold for target detection is σ. l The threshold for tracking and matching is σ. SIoU The confidence threshold for completed tracking is σ. h The shortest tracking step size is t. min The specific steps of the tracking method are as follows:

[0071] (1) Read the hatch detection result D of the next frame. f ;

[0072] (2) Using the threshold σ l Eliminate targets with low confidence levels;

[0073] (3) Traverse set T a All active tracking trajectories t i Find the detection set D of the current frame. f In the middle, with the tracking trajectory t i The largest SIoU (IoU based on position loss information) detected in the previous frame is d. j As d best If the calculated SIoU result is greater than the threshold σ SIoU , then d j Add trajectory t i and from set D f Remove from the middle; if not, then determine t. i Does it satisfy the condition that at least one detection result has a confidence level higher than σ? h Step size greater than t min If the condition is met, then t will be... i Add to set T f Finally, t i From set T a Remove from;

[0074] (4) For other unmatched d j Then, each of these will be created as a new tracking trajectory and added to set T. a middle;

[0075] (5) If not all frames are matched, continue with steps (1)-(5);

[0076] (6) After matching all frames, iterate through set T. a All activated trajectories t i Determine t i Does it satisfy the condition that at least one detection result has a confidence level higher than σ? h Step size greater than t min If the condition is met, then t will be... i Add to set T f If satisfied, t i Add to set T f Finally, t i From set T a Remove from;

[0077] Furthermore, in step (3) of the tracking method, the SIoU loss of the tracking method includes four parts: IoU, angle loss, distance loss, and shape loss, and the specific calculation method is as follows:

[0078] (3.1) IoU:

[0079] IoU is the intersection-union ratio between two bounding boxes A and B. The formula is as follows:

[0080]

[0081] In the formula, Area(A) and Area(B) represent the areas of bounding box A and bounding box B, respectively. The IoU is maximum (1) when the two bounding boxes completely overlap, and minimum (0) when the two bounding boxes do not overlap.

[0082] (3.2) Angular loss Λ:

[0083] As attached Figure 1 As shown, the angle loss Λ describes the minimum angle between the line connecting the center points of bounding boxes A and B and the x and y axes; the formula for calculating the angle loss is:

[0084]

[0085] in,

[0086]

[0087] In the formula, c h σ represents the difference in the vertical positions of the centers of the two hatch boundary frames, σ is the straight-line distance between the centers of the two hatch boundary frames, and x represents the sine value of the small angle α. The angle loss Λ is minimal (0) when the center points are aligned on the x-axis or y-axis, and Λ is maximum (1) when the angle between the connecting line and both the x-axis and the y-axis is 45°.

[0088] (3.3) Distance loss Δ:

[0089] The formula for calculating distance loss Δ is:

[0090]

[0091] in,

[0092] γ=2-Λ

[0093] In the formula, c w This is the difference in the horizontal position of the center of the two hatch boundary frames. Indicates the coordinates of the center point of the hatch boundary frame A. The coordinates of the center point of the hatch boundary frame B are given, and Λ represents the angle loss.

[0094] (3.4) Shape loss Ω:

[0095] The formula for calculating the shape loss Ω is:

[0096]

[0097] in,

[0098]

[0099] In the formula, w A h A The width and height of the hatch boundary frame A are respectively w B h B Define the width and height of the hatch boundary frame A.

[0100] (3.5) Total loss SIoU:

[0101]

[0102] The door tracking method for automatic docking doors of boarding bridges proposed in this patent is compared with the original tracker of YOLOv5. The experimental results are attached. Figure 3 As shown. Figure 3 In Figures (a) and (b), the YOLOv5 tracker demonstrates its tracking performance on the test video; in Figures (c) and (d), the SIoU tracker proposed in this patent demonstrates its tracking performance on the test video. The top left corner of the figure shows the tracker's frame rate in FPS, representing the number of video frames the tracker can process per second. The "door" label on the door detection box in the figure represents the category label output by the YOLOv5 object detection network; the decimal after the category label represents the confidence level of the door detection.

[0103] In terms of tracking frame rate, the YOLOv5 tracker can only reach 15 to 25 frames per second, which is relatively low. The SIoU tracker proposed in this patent can reach 45 to 60 frames per second, which is higher and provides better real-time performance. In terms of tracking accuracy, the YOLOv5 tracker has a detection confidence of only 0.8, and there is a case of missed detection of the hatch in the image (b). The SIoU tracker proposed in this patent has a higher detection confidence of 0.9, and the detection effect in the same frame of image (b) is shown in image (d), with no missed detections, and it can track the hatch more accurately.

Claims

1. A door tracking method for automatic docking doors of boarding bridges, the method comprising a training phase and a tracking phase: During the training phase, a certain number of aircraft door images are collected as training samples, and the training set is augmented using certain geometric transformations and color space transformations. Finally, the training samples are labeled with YOLO format labels, and the improved YOLO door detection model is trained for door detection. During the tracking phase, the camera mounted on the boarding bridge is activated to capture real-time video of the aircraft. The trained door detection model is used to detect the door position in each frame in real time. The tracker uses the tracking information from previous frames to match the door in the current frame with the tracking trajectory of the previous frame and adds the correct tracking target to the tracking trajectory. The specific steps of the training phase are as follows: (1) Use the camera mounted on the boarding bridge to collect images of various aircraft fuselages under different locations, weather and lighting conditions as a dataset, or collect images of aircraft fuselages from the Internet to expand the dataset. (2) Geometric transformation augmentation of images includes: translation, cropping, flipping, rotation, and color transformation augmentation; (3) Label the dataset with YOLO format labels. Each YOLO format label is a txt text file corresponding to each image. Each line of the txt text file corresponds to a hatch in the image. Each line includes the target category, the center coordinates of the target bounding box, and the scale parameter of the bounding box. (4) Split the dataset into a training set and a test set. Input the training set into the improved YOLO object detection network for training. Use the test set to test the detection accuracy and inference speed of the model. When the accuracy and inference speed meet the requirements, use the trained object detection model for hatch detection in the tracking stage. The specific steps of the tracking phase are as follows: (1) Activate the camera mounted on the boarding bridge head to continuously collect video data of the aircraft fuselage; (2) Input the video into the trained hatch detection model, detect the hatch in each frame of the video in real time, and output the detection box of the hatch in the video to the tracker. (3) The tracker combines the tracking trajectory obtained from the previous frame to track the detection box of the current frame; Let the first The output of the frame door detection model is a set , For the set of Each detection result; the activated tracking trajectory is stored in the set. middle, For the first Each detection result; completed tracking trajectories are stored in the collection. The confidence threshold for target detection is... The threshold for tracking and matching is... The confidence threshold for completed tracking is: The shortest tracking step size is ; The specific steps of the tracking method are as follows: (3.1) Read the hatch detection result of the next frame ; (3.2) Using a threshold Eliminate targets with low confidence levels; (3.3) Traversing the set All active tracking trajectories Find the detection set of the current frame. In the middle, with the tracking trajectory Detected in the previous frame , Represents location-based loss information The largest As ;if The calculation result is greater than the threshold. Then Add to track and from the set Remove from the list; otherwise, determine the next step. Does it satisfy the condition that at least one detection result has a confidence level higher than... Step length greater than If the conditions are met, then Add to collection Finally, From the set Remove from; (3.4) For other unmatched Then, each of these will be created as a new tracking trajectory and added to the set. middle; (3.5) If not all frames are matched, continue with steps (3.1)-(3.5); (3.6) After matching all frames, iterate through the set. All activation tracks ,judge Does it satisfy the condition that at least one detection result has a confidence level higher than... Step length greater than If the conditions are met, then Add to collection Finally, From the set Remove from; (4) The boarding bridge adjusts the docking angle and speed according to the tracking trajectory.

2. The door tracking method for automatic docking doors of boarding bridges as described in claim 1, characterized in that, In step (3.3) of the tracking method, the tracking method... The loss consists of four parts: IoU, angle loss, distance loss, and shape loss. The specific calculation method is as follows: （3.3.1） ： The intersection-union ratio (CURRR) between two bounding boxes A and B is calculated using the following formula: ； In the formula, , Let A and B represent the areas of bounding box A and bounding box B, respectively; when the two bounding boxes completely overlap... The maximum value is 1, when the two bounding boxes do not overlap. The minimum value is 0; (3.3.2) Angle loss : The formula for calculating angle loss is: ； in, ； In the formula, This is the difference in the vertical position of the center of the two hatch boundary frames. The straight-line distance between the centers of the two hatch boundary frames. Indicates small angle The sine value, when the center point is aligned on the x-axis or y-axis, the angle loss The minimum is 0. When the angle between the connecting line and both the x and y axes is 45°, the angle loss is 0. The maximum value is 1; (3.3.3) Distance loss : Distance loss The calculation formula is: ； in, ； In the formula, The difference between the horizontal positions of the centers of the two hatch boundary frames, ( , ) represents the coordinates of the center point of the hatch boundary frame A, ( , () represents the coordinates of the center point of the hatch boundary frame B. Indicates angle loss; (3.3.4) Shape loss : Shape loss The calculation formula is: ； in, ； In the formula, , These represent the width and height of the hatch boundary frame A, respectively. , These represent the width and height of the hatch boundary frame B, respectively; (3.3.5) Total Loss : 。

Images