An aircraft door detection method and system based on multi-scale attention

Abstract

The present invention belongs to the field of smart airports, and discloses a method and system for detecting aircraft doors based on multi-scale attention, the method comprising: building a visual Transformer door detection model based on multi-scale attention and deformable attention; collecting an aircraft door data set containing a variety of painting styles, and training a complex painting aircraft door detection model; deploying the trained door detection model on an unmanned boarding bridge, and detecting the images transmitted by the bridgehead camera in real time. The method proposed in the present invention has stronger long-distance information modeling capabilities and global feature extraction capabilities, and is more inclined to identify the door through the shape features of the door, and has higher accuracy in scenarios with complex painting on the fuselage. The model proposed in the present invention adopts an encoder structure based on multi-scale attention with better real-time performance, which can meet the real-time requirements of automatic docking doors of unmanned boarding bridges.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of smart airport technology, and in particular relates to an aircraft door detection method and system based on multi-scale attention. Background Art

[0002] The boarding bridge is a movable passage connecting the terminal and the aircraft at the airport. With the development of the airport and the entire transportation system, the scale of the airport is getting larger and larger, the passenger carrying capacity is also increasing day by day, and the utilization rate of the boarding bridge has also increased significantly. As the link between the airport and the aircraft, the boarding bridge plays a vital role in the operation of the airport. The docking efficiency of the boarding bridge will directly affect the operating efficiency of the airport.

[0003] Manually operated boarding bridges require a lot of training costs and are prone to misoperation, which can lead to airport safety accidents. Unmanned boarding bridges use their own high-precision sensors and intelligent perception algorithms to complete the perception of the surrounding environment and docking with the aircraft door, which can achieve a safer and more efficient door docking process, and is one of the current development trends of smart airports.

[0004] With the development of artificial intelligence and computer vision, YOLO, a real-time target detector based on deep learning, has been applied to the detection of aircraft doors in unmanned boarding bridges, which is the closest implementation scheme to the present invention. YOLO extracts local features in the image of feature aircraft doors, such as door frames, door pedals, handles, observation windows, etc., through a CNN-based backbone network, and fuses multi-scale features with a neck network based on a feature pyramid structure. Finally, door targets of different sizes are detected on feature maps of different scales, and the location information of the target is output through a convolution detection head, which can achieve higher accuracy in the detection of simple painted doors.

[0005] Complex fuselage paint can greatly interfere with cabin door detection. Complex paint has high randomness and contains rich edge information and color information, which seriously affects the local feature extraction of aircraft cabin doors. At present, YOLO can achieve an accuracy rate of more than 90% in the detection of conventionally painted aircraft cabin doors, but it often misses detection on complexly painted aircraft cabin doors, with an accuracy rate of less than 80%.

[0006] Existing aircraft door detection methods use YOLO as the door detection model. YOLO's backbone network, neck network, and detection head all use convolutional architecture. The convolution operation extracts features by weighting the information in the local receptive field. Therefore, YOLO is more inclined to use local information in the image to detect the door. The complex paint on the aircraft door has a great influence on the recognition of the local features of the door. Therefore, YOLO's accuracy is not high in the detection scenario of aircraft doors with complex paint.

[0007] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0008] (1) The existing aircraft door detection model YOLO mainly uses local features in the image to identify the door. When the fuselage has complex paint, the local features of the aircraft door are difficult to extract, and YOLO is not accurate enough in this case.

[0009] (2) Existing Transformer-based target detection models generally suffer from insufficient real-time performance and are unable to meet the real-time requirements of automatic docking doors of unmanned boarding bridges. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides an aircraft door detection method and system based on multi-scale attention.

[0011] The present invention is implemented as follows: a method for detecting aircraft doors based on multi-scale attention, the method specifically comprising:

[0012] S1: Build a visual Transformer door detection model based on multi-scale attention and deformable attention.

[0013] S2: Collect a dataset of aircraft doors with various painting styles and train a detection model for aircraft doors with complex painting styles.

[0014] S3: Deploy the trained door detection model on the unmanned boarding bridge to detect the images transmitted by the bridge head camera in real time.

[0015] Furthermore, the S1 model includes three parts: backbone network, encoder and decoder:

[0016] (1) Backbone network: The backbone network is used to perform preliminary feature extraction on the input door image and perform layer-by-layer downsampling to provide multi-scale feature maps for the neck network. Existing mainstream network models that can output multi-scale feature maps, such as ResNet, MobileNet, and Vision Tranformer, can all be used as backbone networks.

[0017] (2) Encoder: The encoder is an architecture that includes a multi-scale Transformer encoder and a multi-scale feature pyramid. The multi-scale Transformer encoder includes a multi-scale attention module, a feedforward network module, and two residual normalization layers. The multi-scale attention module divides the image into multiple groups by channel. Different groups of feature maps are assigned to different attention heads. Sliding window self-attention operations with different expansion rates are performed on different heads to extract the edge information of door targets of different sizes in the image. The multi-scale feature pyramid receives the output of the multi-scale Transformer encoder and the shallow output of the backbone network, and fuses different multi-scale features through an upsampling path and a downsampling path.

[0018] (3) Decoder: It is used to predict the target from the feature map output by the encoder. The decoder used is a multi-layer deformable Transformer decoder. Each decoder layer consists of a multi-head self-attention module, a deformable attention module and a feedforward network module. The outputs of these modules are residually connected and layer normalized before entering the next module.

[0019] Furthermore, the specific steps of the training phase S2 are as follows:

[0020] (1) Using the camera mounted on the bridgehead, we can collect images of aircraft doors with various paint styles under different locations, weather conditions, and lighting conditions. We can also collect images from the Internet to expand the dataset.

[0021] (2) Perform geometric transformation augmentation on the image, such as translation, cropping, flipping, rotation, and color transformation augmentation, such as brightness change, noise addition, contrast adjustment, saturation adjustment, etc., and expand the number of training samples through data augmentation.

[0022] (3) Label the dataset. Each image corresponds to a txt file. Each line of the txt file corresponds to a hatch in the image. Each line includes the target category, the center coordinates of the target bounding box, and the size.

[0023] (4) Split the data set into a training set and a test set in a certain ratio. Input the training set into the detection model, train the model parameters through the back propagation algorithm, and use the test set to test the detection accuracy and inference speed of the model. When the accuracy and inference speed meet the requirements, the trained target detection model can be used for hatch detection.

[0024] Further, in the detection stage, S3 starts the camera mounted on the bridge head of the boarding bridge to collect real-time video of the fuselage, and uses the trained door detection model to detect the door position in each frame in real time. The specific steps of the detection stage are as follows:

[0025] (1) Deploy the trained model to the industrial computer of the unmanned boarding bridge.

[0026] (2) Start the camera installed at the bridgehead to collect video data of the aircraft door.

[0027] (3) Input the video stream into the trained door detection model, detect the door in each frame of the video in real time, and output the center coordinates, length, and width information of the door detection frame.

[0028] Another object of the present invention is to provide an aircraft door detection system based on multi-scale attention, the system specifically comprising:

[0029] Visual Transformer door detection model, which is based on multi-scale attention and deformable attention, consists of a backbone network, an encoder and a decoder, and is used to extract features and detect aircraft door images.

[0030] The data acquisition device is used to collect images of aircraft doors with various paint styles under different locations, weather and lighting conditions, and to perform geometric and color transformation augmentation on the data set.

[0031] The training module is used to annotate the collected data, split the data set into training set and test set, and train the model parameters through the back propagation algorithm.

[0032] The deployment module is used to deploy the trained door detection model on the industrial computer of the unmanned boarding bridge to achieve real-time detection.

[0033] Furthermore, the visual Transformer door detection model includes:

[0034] The backbone network is used to perform preliminary feature extraction on the input aircraft door image, downsample layer by layer, and output multi-scale feature maps. Mainstream network models such as ResNet, MobileNet, and Vision Transformer can all be used as backbone networks.

[0035] The encoder, including a multi-scale Transformer encoder and a multi-scale feature pyramid, is used to extract edge information of door targets of different sizes in the image through a multi-scale attention module and a feed-forward network module, and fuse multi-scale features through an upsampling path and a downsampling path.

[0036] The decoder consists of a multi-layer deformable Transformer decoder, each layer of which contains a multi-head self-attention module, a deformable attention module, and a feedforward network module, which is used to predict the position and size of the hatch from the feature map output by the encoder.

[0037] Further, the data acquisition device comprises:

[0038] The camera, mounted at the bridgehead, is used to capture images of aircraft doors with various paint styles in different positions, weather and lighting conditions.

[0039] The data augmentation module is used to perform geometric transformation (such as translation, cropping, flipping, rotation) and color transformation (such as brightness transformation, noise addition, contrast adjustment, saturation adjustment) on the collected images to expand the number of training samples.

[0040] The data annotation module is used to annotate the data set. Each image records the category, center coordinates, and size of the hatch.

[0041] Further, the deployment module includes:

[0042] Industrial computers are used to deploy the trained door detection model and process real-time video streams.

[0043] The real-time detection module starts the camera installed at the bridgehead of the boarding bridge, collects video data of the aircraft door, inputs the video stream into the trained door detection model, detects the door in each frame of the video in real time, and outputs the center coordinates, length and width information of the door detection frame.

[0044] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0045] First, the method proposed in the present invention has stronger long-distance information modeling capabilities and global feature extraction capabilities, and is more inclined to identify cabin doors through their shape features, and has higher accuracy in scenarios where the fuselage has complex paint. When dealing with aircraft cabin doors with complex paint, the traditional YOLO model is easily disturbed by complex paint because it relies on local information, resulting in many false detections and missed detections of aircraft cabin doors with complex paint. The present invention effectively models long-distance dependencies in images by introducing a multi-scale attention mechanism and a visual Transformer architecture, thereby enhancing the ability to recognize cabin door shape features, thereby significantly improving detection accuracy in complex paint scenarios.

[0046] In addition, the model proposed in the present invention adopts a multi-scale attention-based encoder structure with better real-time performance, which can meet the real-time requirements of the automatic docking door of the unmanned boarding bridge and ensure the operating efficiency of the automatic docking system of the boarding bridge.

[0047] The present invention proposes a method for detecting aircraft doors based on multi-scale attention, and designs and optimizes the door detection model structure for complex painted aircraft door detection scenarios. The model proposed in the present invention adopts a visual Transformer architecture based on multi-scale attention, which has stronger long-distance information modeling capabilities than YOLO, and is more inclined to identify doors by the shape features of doors in images, which is conducive to improving the accuracy of complex painted aircraft door detection and reducing false detections and missed detections.

[0048] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0049] With the continuous advancement of smart airport construction, unmanned boarding bridges will become an important trend in future airport operations, and their market demand will continue to expand. The technical solution of the present invention can significantly improve the accuracy and stability of unmanned boarding bridges in the detection of complex painted aircraft doors, reduce false detection and missed detection rates, and thus improve the efficiency and safety of the entire docking process. This will save a lot of manpower and training costs for airport operators, improve the operating efficiency and service quality of the airport, and bring significant economic benefits.

[0050] The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0051] Since the aircraft's paint style is random, the color and texture information is complex, and the fuselage paint often changes, it is a difficult feature for the door detection model to learn, and the complex paint causes serious interference to the recognition of the aircraft door. For a long time, the detection of aircraft doors with complex paint has been a difficult problem for the application of the technology of automatic docking of aircraft doors with unmanned boarding bridges. The technical solution of the present invention improves the detection accuracy in the scenario of aircraft doors with complex paint by optimizing the model structure, solves this technical problem that has long plagued the industry, effectively reduces the restrictions on the use scenarios of unmanned boarding bridges, promotes the application of unmanned boarding bridges, and injects new vitality into the development of smart airports.

[0052] Third, the present invention aims to solve the problems of low accuracy, poor adaptability and difficulty in achieving real-time detection in complex environments in the prior art of aircraft door detection, and proposes an aircraft door detection method based on multi-scale attention, which significantly improves the accuracy and efficiency of detection. Traditional methods often rely on a single image processing technology or a simple machine learning model. When faced with different painting styles, lighting conditions and complex backgrounds, the detection effect is not ideal, resulting in inaccurate or even failed docking of boarding bridges, which seriously affects the operating efficiency and safety of airports.

[0053] First, by constructing a visual Transformer detection model with multi-scale attention and deformable attention, the present invention can effectively extract multi-scale features, especially when dealing with complex backgrounds and diverse paint jobs. The combination of the multi-scale Transformer encoder and decoder enables the model to accurately identify doors of different sizes and shapes and extract edge information of the doors, solving the problem of inaccurate detection caused by insufficient feature extraction in traditional methods. In addition, by introducing a deformable attention mechanism, the model of the present invention can flexibly adjust the attention window within the deformation area, more accurately locate the specific position and size of the door, and further improve the accuracy of detection.

[0054] Secondly, the present invention collects and trains data sets under various painting styles and complex lighting conditions, so that the model can adapt to a variety of real-world scenarios. In particular, under low-light conditions at night, traditional detection methods are prone to failure due to insufficient light. The present invention ensures the robustness of detection under various lighting conditions through special processing of low-light data and tuning of the model. This advancement solves the problem of insufficient adaptability of traditional detection systems in specific environments, allowing hatch door detection to operate efficiently in all weather and all scenarios.

[0055] Finally, the present invention also achieves efficient processing of real-time camera transmission images through efficient model training and optimization. After the trained model is deployed to the unmanned boarding bridge, the door information in the video stream can be analyzed and detected in real time, and the detection results can be quickly output. Compared with traditional methods, the present invention not only achieves significant improvements in detection accuracy, but also greatly improves the detection speed, can respond to the complex operational needs of the airport in a timely manner, and significantly improves the airport's automation level and operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of an aircraft door detection method based on multi-scale attention provided by an embodiment of the present invention.

[0057] Figure 2 It is a general structural diagram of the model provided by an embodiment of the present invention.

[0058] Figure 3 It is a structural diagram of an encoder provided by an embodiment of the present invention.

[0059] Figure 4 It is a structural diagram of a multi-scale Transformer encoder provided in an embodiment of the present invention.

[0060] Figure 5 It is a module diagram of an aircraft door detection system based on multi-scale attention provided by an embodiment of the present invention.

[0061] Figure 6It is a rendering of an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] The aircraft door detection method based on multi-scale attention provided by the embodiment of the present invention builds a visual Transformer detection model with multi-scale attention and deformable attention, and performs real-time video detection after training with an aircraft door dataset containing various painting styles, thereby achieving efficient and accurate detection of aircraft doors. The specific working principle is as follows:

[0064] 1. Model building.

[0065] Step S1: Build a visual Transformer door detection model based on multi-scale attention and deformable attention.

[0066] Backbone network: used to extract preliminary features from the input aircraft door image. The backbone network downsamples layer by layer and outputs multi-scale feature maps, which are applicable to a variety of mainstream network models such as ResNet, MobileNet, Vision Transformer, etc.

[0067] Encoder: It consists of a multi-scale Transformer encoder and a multi-scale feature pyramid. The multi-scale Transformer encoder divides the image into multiple feature groups through a multi-scale attention module, a feed-forward network module, and a residual normalization layer. Different groups perform sliding window self-attention operations with different expansion rates on different attention heads to extract edge information of door targets of different sizes. The multi-scale feature pyramid fuses multi-scale features.

[0068] Decoder: It consists of a multi-layer deformable Transformer decoder. Each layer contains a multi-head self-attention module, a deformable attention module, and a feedforward network module. Through residual connections and layer normalization, it predicts the position and size of the target door.

[0069] 2. Data set collection and model training.

[0070] Step S2: Collect a data set of aircraft doors with various painting styles and train a complex painting aircraft door detection model.

[0071] Data collection: Using cameras mounted on the bridgehead, we collected images of aircraft doors with various paint styles in different locations, weather, and lighting conditions to expand the data set.

[0072] Data augmentation: Perform geometric transformation (translation, cropping, flipping, rotation) and color transformation (brightness transformation, noise addition, contrast adjustment, saturation adjustment) on the collected images to expand the number of training samples.

[0073] Data annotation: Annotate the data set. Each image corresponds to a txt text file, which records the category, center coordinates and size of each hatch.

[0074] Model training: Split the data set into training and test sets in proportion, input the training set into the detection model, and train the model parameters through the back propagation algorithm. Use the test set to evaluate the detection accuracy and inference speed of the model to ensure that the model performance meets the requirements.

[0075] 3. Model deployment and real-time detection.

[0076] Step S3: Deploy the trained door detection model on the unmanned boarding bridge to detect the images transmitted by the bridgehead camera in real time.

[0077] Model deployment: Deploy the trained model to the industrial computer of the unmanned boarding bridge.

[0078] Video acquisition: Start the camera at the bridgehead of the boarding bridge to collect video data of the aircraft door in real time.

[0079] Real-time detection: Input the video stream into the trained door detection model, detect the door in each frame of the video in real time, and output the center coordinates, length and width information of the door detection frame.

[0080] like Figure 1 As shown, an embodiment of the present invention provides an aircraft door detection method based on multi-scale attention, and the method specifically includes:

[0081] S1: Build a visual Transformer door detection model based on multi-scale attention and deformable attention.

[0082] S2: Collect a dataset of aircraft doors with various painting styles and train a detection model for aircraft doors with complex painting styles.

[0083] S3: Deploy the trained door detection model on the unmanned boarding bridge to detect the images transmitted by the bridge head camera in real time.

[0084] The following are two specific embodiments based on the above technical solution.

[0085] Example 1: Aircraft door inspection under complex painting conditions.

[0086] At a large international airport, boarding bridges need to quickly dock aircraft from different airlines, which often have different paint designs. Due to the complex and varied paint styles, traditional door detection methods have difficulty accurately identifying the door position under complex backgrounds.

[0087] 1. Dataset collection and processing.

[0088] Cameras were installed on multiple boarding bridges at the airport to collect a large number of door images of aircraft with different paint styles at different time periods and weather conditions. The dataset includes aircraft of different brands and models to ensure that a variety of complex paint designs are covered.

[0089] Data augmentation technology was used to translate, rotate, flip, and add noise to the original images, further expanding the data set.

[0090] 2. Model training.

[0091] ResNet is used as the backbone network, and the Transformer structure with multi-scale attention and deformable attention is combined to perform feature extraction, encoding and decoding of images.

[0092] The processed data set was split into 80% training set and 20% test set, and the model was trained using this data. Through multiple rounds of iterations, the detection accuracy of the model continued to improve, and eventually reached 96% accuracy on the test set, meeting the actual application needs of the airport.

[0093] 3. Model deployment and application.

[0094] The trained model is deployed on the unmanned boarding bridge to monitor the images transmitted by the camera in real time. Every time the aircraft docks, the system can accurately detect the position of the cabin door within 1 second and automatically dock with the boarding bridge, greatly improving the docking efficiency.

[0095] The system performs stably under different weather conditions (such as strong sunlight, cloudy days, and night), effectively improving the airport's automation level.

[0096] Embodiment 2: Aircraft door detection under low light conditions at night.

[0097] During nighttime operations at some airports, due to insufficient lighting conditions, the accuracy of traditional door detection methods is significantly reduced, which can easily lead to inaccurate docking of boarding bridges, affecting flight scheduling and safety.

[0098] Implementation steps:

[0099] 1. Dataset collection and processing.

[0100] Specifically for night conditions, a large number of low-light hatch images were collected. In order to enhance the robustness of the model under night conditions, the collection of low-light data was increased, covering a variety of paint styles and images under different lighting conditions.

[0101] During data augmentation, brightness and contrast were specifically adjusted to simulate different light intensities and light source positions.

[0102] 2. Model training.

[0103] The backbone network uses Vision Transformer, combined with a multi-scale attention mechanism, to extract subtle features in the image and accurately locate doors of different sizes and shapes through a deformable attention module.

[0104] During the training process, the model’s detection accuracy in low-light conditions was further improved by adjusting the learning rate and the weight of the loss function. Ultimately, the model’s detection accuracy in low-light environments reached 93%.

[0105] 3. Model deployment and application.

[0106] The model is deployed on a jet bridge equipped with an infrared camera to handle door detection tasks at night in real time. Regardless of the lighting conditions, the system can accurately identify the door position and ensure accurate docking of the jet bridge.

[0107] Through comparative tests, it was found that after deploying the system, the time for night boarding bridge docking was shortened by 30%, and safety and efficiency were significantly improved.

[0108] These two embodiments demonstrate the application of the present invention in aircraft door detection under complex painting and low-light conditions at night, verify the robustness and practicality of the system, and provide reliable technical support for airport automation operations.

[0109] The S1 model includes three parts: backbone network (Backbone), encoder (Encoder), and decoder (Decoder). The model structure is as follows Figure 2 As shown:

[0110] (1) Backbone network.

[0111] The backbone network is used to perform preliminary feature extraction on the input door image and perform layer-by-layer downsampling to provide a multi-scale feature map for the neck network. In the model proposed by the present invention, existing mainstream network models that can output multi-scale feature maps, such as ResNet, MobileNet, Vision Transformer, etc., can be used as the backbone network.

[0112] (2) Encoder.

[0113] In the model proposed by this invention, the design of the encoder is optimized for the complex painted aircraft door detection scenario. The overall structure of the encoder used in this model is as follows Figure 3 As shown in Figure 1, it is an architecture that includes a multi-scale Transformer encoder and a multi-scale feature pyramid. The multi-scale Transformer encoder receives feature maps from the deepest layer of the backbone network, which has rich semantic information and has the smallest feature map size, reducing the computational redundancy of the encoder. The multi-scale feature pyramid receives the output of the multi-scale Transformer encoder and the shallow output of the backbone network, and fuses different multi-scale features through an upsampling path and a downsampling path.

[0114] The multi-scale Transformer encoder is an important component of the network. It can effectively model long-range information, which is crucial for capturing the shape characteristics of aircraft doors. The structure of the multi-scale Transformer encoder is as follows: Figure 4 As shown in the figure, it includes a multi-scale attention module, a feedforward network module and two residual normalization layers. The multi-scale attention module is optimized for the detection of aircraft doors with complex paint. The module divides the image into multiple groups by channel, assigns different groups of feature maps to different attention heads, and performs sliding window self-attention operations with different expansion rates on different heads to extract the edge information of door targets of different sizes in the image. The main feature of an aircraft door is the shape feature of the door edge. The expansion attention operation can skip the pixels around the query and perform scaled dot product attention with the key-value pairs of distant pixels, which is conducive to reducing the local feature interference caused by complex painting in the image, modeling long-distance dependencies, and improving the ability to extract door shape features.

[0115] (3) Decoder.

[0116] In the model proposed in the present invention, the decoder is used to predict the target from the feature map output by the encoder. The decoder used is a multi-layer deformable Transformer decoder. Each decoder layer consists of a multi-head self-attention module, a deformable attention module and a feedforward network module, and the outputs of these modules are residually connected and layer normalized before entering the next module.

[0117] The shapes of aircraft doors are different from different perspectives. The decoder uses the dynamic spatial sampling capability of deformable attention to effectively capture the shape features of targets of different shapes and improve the accuracy of the model's detection of aircraft doors from different perspectives.

[0118] In the S2, during the training phase, a certain number of conventionally painted and complexly painted aircraft door images are collected as training samples, and the training set is augmented by geometric transformation and color transformation. Finally, labels are annotated for the training samples to train the door detection model.

[0119] The specific steps of the training phase are as follows:

[0120] (1) Using the camera mounted on the bridgehead, we can collect images of aircraft doors with various paint styles under different locations, weather conditions, and lighting conditions. We can also collect images from the Internet to expand the dataset.

[0121] (2) Perform geometric transformation augmentation on the image, such as translation, cropping, flipping, and rotation, as well as color transformation augmentation, such as brightness change, noise addition, contrast adjustment, and saturation adjustment. By expanding the number of training samples through data augmentation, the accuracy and robustness of the door detection model are improved.

[0122] (3) Label the dataset. Each image corresponds to a txt file. Each line of the txt file corresponds to a hatch in the image. Each line includes the target category, the center coordinates of the target bounding box, and the size.

[0123] (4) Split the data set into a training set and a test set according to a certain ratio. Input the training set into the detection model, train the model parameters through the back propagation algorithm, and use the test set to test the detection accuracy and inference speed of the model. When the accuracy and inference speed meet the requirements, the trained target detection model can be used for hatch detection.

[0124] In the detection phase, the camera mounted on the boarding bridge is started to collect real-time video of the fuselage, and the door position in each frame is detected in real time using the trained door detection model. The specific steps of the detection phase are as follows:

[0125] (1) Deploy the trained model to the industrial computer of the unmanned boarding bridge.

[0126] (2) Start the camera installed at the bridgehead to collect video data of the aircraft door.

[0127] (3) Input the video stream into the trained door detection model, detect the door in each frame of the video in real time, and output the center coordinates, length, and width information of the door detection frame.

[0128] like Figure 5 As shown, an aircraft door detection system based on multi-scale attention provided by an embodiment of the present invention specifically includes:

[0129] The input module uses the camera installed at the bridgehead of the boarding bridge to collect video in real time.

[0130] The feature extraction module extracts and fuses features at multiple scales through a visual Transformer door detection model based on multi-scale attention and deformable attention.

[0131] The detection module is used to detect the hatch in each frame of video in real time.

[0132] The inspection effect of complex painted hatch is shown below: Figure 6 It can be seen that the model proposed in the embodiment of the present invention can achieve high-precision real-time detection of complex painted aircraft doors.

[0133] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0134] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for detecting aircraft doors based on multi-scale attention, characterized in that: The method includes: S1: Build a visual Transformer door detection model based on multi-scale attention and deformable attention; S2: Collect a dataset of aircraft doors with various painting styles and train a detection model for aircraft doors with complex painting styles. S3: Deploy the trained door detection model on the unmanned boarding bridge to detect the images transmitted by the bridge head camera in real time; The S1, visual Transformer door detection model based on multi-scale attention and deformable attention includes three parts: backbone network, encoder, and decoder: (1) Backbone network: The backbone network is used to perform preliminary feature extraction on the input door image and perform layer-by-layer downsampling to provide multi-scale feature maps for the neck network. Mainstream network models that can output multi-scale feature maps are used as the backbone network, including ResNet, MobileNet, and Vision Tranformer. (2) Encoder: The encoder is an architecture that includes a multi-scale Transformer encoder and a multi-scale feature pyramid. The multi-scale Transformer encoder includes a multi-scale attention module, a feedforward network module, and two residual normalization layers. The multi-scale attention module divides the image into multiple groups by channel. Different groups of feature maps are assigned to different attention heads. Sliding window self-attention operations with different expansion rates are performed on different heads to extract edge information of door targets of different sizes in the image. The multi-scale feature pyramid receives the output of the multi-scale Transformer encoder and the shallow output of the backbone network, and fuses different multi-scale features through an upsampling path and a downsampling path. (3) Decoder: It is used to predict the target from the feature map output by the encoder. The decoder used is a multi-layer deformable Transformer decoder. Each decoder layer consists of a multi-head self-attention module, a deformable attention module and a feedforward network module. The outputs of these modules are residually connected and layer normalized before entering the next module.

2. The aircraft door detection method based on multi-scale attention as claimed in claim 1, characterized in that: The specific steps of the S2 training phase are as follows: (1) Using cameras mounted on the bridgehead, collect images of aircraft doors with various paint styles under different locations, weather, and lighting conditions, or collect images from the Internet to expand the dataset; (2) Perform geometric transformation augmentation on the image, including translation, cropping, flipping, and rotation; perform color transformation augmentation on the image, including brightness change, adding noise, contrast adjustment, and saturation adjustment, and expand the number of training samples through data augmentation; (3) Label the dataset. Each image corresponds to a txt file. Each line of the txt file corresponds to a hatch in the image. Each line includes the target category, the center coordinates of the target bounding box, and the size. (4) Split the data set into a training set and a test set in a certain ratio. Input the training set into the detection model, train the model parameters through the back propagation algorithm, and use the test set to test the detection accuracy and inference speed of the model. When the accuracy and inference speed meet the requirements, the trained target detection model is used for hatch detection.

3. The aircraft door detection method based on multi-scale attention as claimed in claim 1, characterized in that: In the detection phase, the camera mounted on the bridge head of the boarding bridge is started to collect real-time video of the fuselage, and the door position in each frame is detected in real time using the trained door detection model. The specific steps of the detection phase are as follows: (1) Deploy the trained model to the industrial computer of the unmanned boarding bridge; (2) Start the camera installed at the bridgehead to collect video data of the aircraft door; (3) Input the video stream into the trained door detection model, detect the door in each frame of the video in real time, and output the center coordinates, length, and width information of the door detection frame.

4. An aircraft door detection system based on multi-scale attention according to any one of the methods of claims 1-3, characterized in that: The system specifically includes: The visual Transformer door detection model, which is based on multi-scale attention and deformable attention and includes a backbone network, an encoder, and a decoder for feature extraction and detection of aircraft door images; A data acquisition device for collecting images of aircraft doors with various paint styles at different locations, weather and lighting conditions, and augmenting the data set with geometric and color transformations; The training module is used to annotate the collected data, split the data set into training set and test set, and train the model parameters through the back propagation algorithm; The deployment module is used to deploy the trained door detection model on the industrial computer of the unmanned boarding bridge to achieve real-time detection.

5. The system according to claim 4, characterized in that The visual Transformer door detection model includes: The backbone network is used to perform preliminary feature extraction on the input aircraft door image, downsample layer by layer, and output multi-scale feature maps. Mainstream network models are used as the backbone network, including ResNet, MobileNet, and Vision Transformer. An encoder, including a multi-scale Transformer encoder and a multi-scale feature pyramid, is used to extract edge information of door objects of different sizes in the image through a multi-scale attention module and a feed-forward network module, and fuse multi-scale features through an upsampling path and a downsampling path; The decoder consists of a multi-layer deformable Transformer decoder, each layer of which contains a multi-head self-attention module, a deformable attention module, and a feedforward network module, which is used to predict the position and size of the hatch from the feature map output by the encoder.

6. The system according to claim 4, characterized in that The data acquisition device comprises: Cameras, mounted at the bridgehead, are used to capture images of aircraft doors in various paint styles in different locations, weather and lighting conditions; The data augmentation module is used to perform geometric transformation and color transformation augmentation on the collected images to expand the number of training samples. Geometric transformation includes translation, cropping, flipping, and rotation; color transformation includes brightness transformation, adding noise, contrast adjustment, and saturation adjustment. The data annotation module is used to annotate the data set. Each image records the category, center coordinates, and size of the hatch.

7. The system according to claim 4, characterized in that The deployment module includes: Industrial computers, used to deploy trained door detection models and process real-time video streams; The real-time detection module starts the camera installed at the bridgehead of the boarding bridge, collects video data of the aircraft door, inputs the video stream into the trained door detection model, detects the door in each frame of the video in real time, and outputs the center coordinates, length and width information of the door detection frame.

Images