Aircraft Door Pose Detection Method Based on Fuzzy Fusion and Automatic Berthing Control Method for Boarding Bridges

Abstract

The present invention relates to a method for detecting the position and attitude of an aircraft cabin door based on fuzzy fusion and a method for automatically controlling the approach of a boarding bridge to the aircraft. It solves the technical problems of poor docking accuracy and poor environmental adaptability of the existing boarding bridge when docking with the aircraft cabin door, inaccurate positioning of the aircraft cabin door, poor stability, and poor environmental adaptability. During the cabin door detection process, first, two types of cabin door position detection algorithms based on image processing and deep learning are respectively constructed, and then the detection results of the two different types are fused through a fuzzy logic reasoning method to obtain the fused cabin door position information. Finally, the boarding bridge is controlled to dock with the cabin door based on the fused cabin door position information.

Description

Technical Field

[0001] The present invention relates to the technical field of civil aviation airport boarding bridges, and more specifically, to a method for detecting the pose of an aircraft cabin door based on fuzzy fusion and a method for automatically controlling the boarding bridge to approach the aircraft. Background Art

[0002] In recent years, with the rapid development of the global aviation industry, traditional airports have gradually transformed into intelligent airports to meet challenges such as growing transportation demands and service pressures. Intelligent airports comprehensively introduce automation and intelligent technologies to optimize and improve the operation efficiency and quality of airports.

[0003] As a key device connecting the terminal building and the aircraft, the digitization, automation, and intelligence of the boarding bridge during the docking process with the aircraft are the research directions of professionals in this field.

[0004] Currently, the docking of the boarding bridge with the aircraft relies on manual operation. The operator controls the boarding bridge and completes the docking visually, which takes about 3 minutes. However, the boarding bridge docking method based on manual operation has the following drawbacks: the operation process is complex and cumbersome; due to visual estimation errors and the motion inertia of the boarding bridge, it is difficult to ensure docking accuracy, and accidents are prone to occur under adverse weather conditions or during fatigue driving; it has high requirements for the technical expertise of operators, resulting in high training costs and long training cycles for airports. To address the above drawbacks, professionals in this field have developed some intelligent boarding bridge automatic docking systems based on vision technology or sensor technology, such as the patent application with the publication number CN 115783294A, the patent application with the publication number CN111522345A, the patent application with the publication number CN118135356A, the patent with the authorization announcement number CN111776243B, and the patent with the authorization announcement number CN 112034831B. However, the use of a single detection method (vision technology or sensor technology) has poor robustness under different working conditions, and the docking accuracy of these automatic docking systems needs to be improved, with poor environmental adaptability, inaccurate positioning of the aircraft cabin door, poor stability, and poor environmental adaptability. Summary of the Invention

[0005] The present invention aims to solve the technical problems of poor docking accuracy, poor environmental adaptability, inaccurate positioning of the aircraft cabin door, poor stability, and poor environmental adaptability when the existing boarding bridge docks with the aircraft cabin door, and provides a method for detecting the pose of an aircraft cabin door based on fuzzy fusion and a method for automatically controlling the boarding bridge to approach the aircraft.

[0006] The key to the intelligent boarding bridge automatic docking system lies in the position detection of the aircraft cabin door. Most of the existing technologies adopt a single detection method, namely traditional image processing and deep learning-based methods. Although these two detection methods perform well separately under good working conditions, their robustness is not strong under different working conditions.

[0007] The present invention provides a method for detecting the position and pose of an aircraft cabin door based on fuzzy fusion, including the following steps:

[0008] Step 1, obtain the position information of the cabin door through an image processing algorithm;

[0009] For the image containing the cabin door collected by the binocular camera, use an edge detection algorithm to obtain the contour of the cabin door;

[0010] Obtain the minimum circumscribed rectangle of the cabin door contour to represent the outer contour of the cabin door;

[0011] Judge the position and pose of the minimum circumscribed rectangle according to the aspect ratio of the minimum circumscribed rectangle to obtain the direction of the minimum circumscribed rectangle. When the aspect ratio is greater than 1, it is positive. In the case where the minimum circumscribed rectangle is positive, obtain the pixel coordinates D 1L (u 1 l ,v 1 l ) and / or the pixel coordinates D 1R (u 1 r ,v 1 r ) of the lower right corner point;

[0012] Step 2, obtain the position information of the cabin door through the YOLOv5 detection algorithm;

[0013] Establish a YOLOv5 object detection model through the training data set;

[0014] Input the image containing the cabin door collected by the binocular camera into the YOLOv5 object detection model. The YOLOv5 object detection model outputs the detection result, identifies the cabin door. In the detection result image, the cabin door is surrounded by a detection frame. Obtain the pixel coordinates D 2L (u 2 l ,v 2 l ) of the lower left corner point of the detection frame, and / or the pixel coordinates D 2R (u 2 r ,v 2 r ) of the lower right corner point;

[0015] Step 3, obtain the optimized cabin door position information through the fuzzy fusion algorithm;

[0016] Step (1), determine the system input and output variables;

[0017] Multiple input variables of the system are determined according to the actual working conditions of the boarding bridge docking the aircraft cabin door. The two output variables of the system are the fusion weights w 1 and the fusion weight w 2 ;

[0018] Step (2), fuzzification of the input and output variables;

[0019] Step (3), establish a fuzzy rule base;

[0020] Step (4), fusion processing;

[0021] According to the fusion weights w 1 and w 2 , perform fusion processing on the cabin door position information obtained in step 1 and the cabin door position information obtained in step 2 to obtain the optimized cabin door position information F L and / or F R , F L The calculation formula of is as follows:

[0022] F L = w 1 ·D 1L + w 2 ·D 2L

[0023] where, D 1L is the pixel coordinate of the lower left corner point obtained in step 1, and D 2L is the pixel coordinate of the lower left corner point obtained in step 2;

[0024] F R The calculation formula of is as follows:

[0025] F R = w 1 ·D 1R + w 2 ·D 2R

[0026] where, D 1R is the pixel coordinate of the lower right corner point obtained in step 1, and D 2R is the pixel coordinate of the lower right corner point obtained in step 2.

[0027] Preferably, in step 1, the four vertices of the minimum bounding rectangle are sorted in descending order of the y coordinate to obtain two points with larger y coordinates as the two lower corner points, and then the two lower corner points are sorted in ascending order of the x coordinate to obtain the lower left corner point and the lower right corner point respectively. Furthermore, the pixel coordinate of the lower left corner point is D 1L (u1 l , v 1 l ) and / or the pixel coordinates D of the lower right corner point 1R (u 1 r , v 1 r ).

[0028] Preferably, the four vertices of the detection box are sorted in descending order of the y coordinate to obtain two points with larger y coordinates as the two lower corner points, and then the two lower corner points are sorted in ascending order of the x coordinate to obtain the lower left corner point and the lower right corner point respectively, and further obtain that the pixel coordinates of the lower left corner point are D 2L (u 2 l , v 2 l ) and / or the pixel coordinates D of the lower right corner point 2R (u 2 r , v 2 r ).

[0029] Preferably, in step 1, the contour of the hatch is obtained by using an edge detection algorithm. First, the image containing the hatch is grayscale processed to obtain a grayscale image, and the high-dimensional color information is reduced in dimension and converted into one-dimensional pixel point intensity information; secondly, the grayscale image is processed by the Canny edge detection algorithm to obtain edges; then, the edges that conform to the shape of the hatch are extracted, that is, the contour of the hatch is obtained.

[0030] Preferably, in step 2, a CBAM module is added to the YOLOv5 object detection model to obtain an improved YOLOv5 object detection model, and the improved YOLOv5 object detection model is used for the recognition and detection of the hatch.

[0031] Preferably, in step (1) of step 3:

[0032] The multiple input variables of the system are respectively the state of the boarding bridge, the illumination intensity of the environment, the distance, and the detection perspective;

[0033] The fuzzy levels of the boarding bridge state are divided into: static and dynamic; the fuzzy levels of the illumination intensity are divided into: strong light, natural light, and weak light; the fuzzy levels of the distance are divided into: short distance, medium distance, and long distance; the fuzzy levels of the perspective are divided into: front view and oblique view; the fusion weight w 1 of the fuzzy set and the fusion weight w 2 have the same fuzzy level division, both including three parameters: low weight, medium weight, and high weight.

[0034] Preferably, in step 2, images under different lighting conditions and different angles are taken respectively when constructing the training dataset.

[0035] Preferably, according to the optimized hatch position information F L and / or F R , combined with the depth information obtained based on the binocular camera ranging principle, the three-dimensional spatial information F L (X L , Y L , Z L ) and / or F R (X R , Y R , Z R ) is obtained.

[0036] The present invention also provides a method for automatically controlling the boarding bridge to approach the aircraft using the three-dimensional spatial information F L (X L , Y L , Z L ) and / or F R (X R , Y R , Z R ), comprising the following steps:

[0037] Step 1), establishing a motion model of the boarding bridge;

[0038] The boarding bridge includes a turntable column, a turntable, an inner channel, an outer channel, a lifting and walking assembly, and the front end of the aircraft receiving port. There is also an aircraft receiving port front-end horizontal rotation driving mechanism between the outer channel and the front end of the aircraft receiving port. The lifting and walking assembly is provided with differential wheels for controlling the boarding bridge to move forward, backward, and turn; the lifting and walking assembly is also provided with a lifting mechanism;

[0039] The control quantities for the actions of the lifting and walking assembly are the traveling distance of the differential wheels, the rotation angle of the differential wheels, and the lifting height of the lifting mechanism. The control quantity for the action of the aircraft receiving port front-end horizontal rotation driving mechanism is the horizontal rotation angle;

[0040] Taking the turntable column as the base coordinate system (X 0 , Y 0 , Z 0 ), the coordinate system where the center of the turntable is located is (X 1 , Y 1 , Z 2 ), the coordinate system where the connection between the inner channel and the turntable is located is (X 2 , Y 2 , Z 2 ), and the coordinate system where the center of the differential wheels in the lifting and walking assembly is located is (Z w , Y w , Z w), the coordinate system where the center point of the floor at the front end of the boarding gate pick-up area is located is (X c , Y c , Z c );

[0041] Establish a motion model of the boarding bridge through forward kinematics;

[0042] Step 2), solve the target value of the control quantity through inverse kinematics;

[0043] Based on the current value of the traveling distance of the differential wheels of the lifting and traveling assembly, the current value of the rotation angle of the differential wheels, the current value of the lifting height of the lifting mechanism, the current value of the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the boarding gate pick-up area, the three-dimensional coordinates of the docking target point and the current position information of the front end of the boarding gate pick-up area, and based on the principle of inverse kinematics, reverse solve the target value of the traveling distance of the differential wheels, the target value of the rotation angle of the differential wheels, the target value of the lifting height of the lifting mechanism, and the target value of the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the boarding gate pick-up area through the motion model of the boarding bridge;

[0044] Step 3), generate a motion instruction according to the target value of the traveling distance of the differential wheels, the target value of the rotation angle of the differential wheels, the target value of the lifting height of the lifting mechanism, and the target value of the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the boarding gate pick-up area, and send it to the corresponding actuator in the boarding bridge to make the actuator act and then perform the docking operation.

[0045] The beneficial effects of the present invention are as follows: Two types of hatch position detection algorithms based on image processing and deep learning are respectively constructed, and then the detection results of the two different types are fused through the fuzzy logic reasoning method, that is, by assigning different weights to the two detection results under different working conditions, so as to obtain the fused detection result, thereby improving the robustness of the hatch position detection under different working conditions on the basis of ensuring real-time performance, making the positioning of the aircraft hatch more accurate, stable, and adaptable to the environment. Using the fused hatch position detection result to control the boarding bridge to complete the docking operation improves the docking accuracy, enhances the environmental adaptability, and also improves the docking operation efficiency.

[0046] The further features and aspects of the present invention will be clearly recorded in the following description of the specific embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic structural diagram of the boarding bridge;

[0048] Figure 2 is a flowchart of the method for detecting the position of the aircraft hatch based on fuzzy fusion of the present invention;

[0049] Figure 3 is a flowchart of obtaining the contour of the hatch using the Canny edge detection algorithm;

[0050] Figure 4 It is a schematic diagram of the minimum circumscribed rectangle of the hatch contour obtained after the hatch is acquired through an image processing algorithm;

[0051] Figure 5 It is the architecture diagram of the CBAM module;

[0052] Figure 6 It is a schematic structural diagram of a binocular camera and a ranging sensor installed at the front end of the boarding bridge's receiving port;

[0053] Figure 7 It is a flowchart of obtaining the optimized hatch position information through a fuzzy fusion algorithm;

[0054] Figure 8 It is a schematic diagram of forming various coordinate systems based on the boarding bridge;

[0055] Figure 9 It is a schematic diagram of selecting a docking point on the floor at the front end of the boarding bridge's receiving port;

[0056] Figure 10 It is a schematic diagram of selecting a docking point on the floor at the front end of the boarding bridge's receiving port.

[0057] Explanation of symbols in the figure:

[0058] 1. Front end of the receiving port, 1-1. Floor, 1-1-1. Central axis, 2. Binocular camera, 4. First ranging sensor, 5. Second ranging sensor; 6. Turntable column, 7. Turntable, 8. Inner channel, 9. Outer channel, 10. Lifting and walking assembly, 11. Rubber cylinder. Detailed implementation manner

[0059] The following further elaborates on the present invention with specific embodiments with reference to the accompanying drawings.

[0060] Reference Figure 1 and 6 , the binocular cameras 2 are respectively installed at the front end 1 of the boarding bridge's receiving port. The first ranging sensor 4 and the second ranging sensor 5 are respectively installed on the left and right sides at the bottom of the front end 1 of the receiving port.

[0061] The control method for the boarding bridge to dock with the hatch disclosed in the present invention mainly includes the following steps:

[0062] The first step is to collect images of the area where the hatch is located through the binocular cameras 2.

[0063] The second step is to obtain the position information of the hatch.

[0064] Two kinds of position information of the hatch are obtained through an image processing algorithm and a deep learning algorithm respectively, and then the two kinds of position information are fused. The fusion process is to assign different weights to the two kinds of position information under different working conditions to obtain the optimized position information of the hatch, which improves the robustness of the hatch detection algorithm under different working conditions on the basis of ensuring real-time performance and is beneficial to enhancing the environmental adaptability.

[0065] Step 1: Obtain the position information of the hatch through an image processing algorithm.

[0066] In the image containing the hatch collected by the camera, since the hatch usually presents edge features different from the surrounding background, it is more appropriate to use an edge detection algorithm to detect the hatch. The edge detection algorithm can identify the boundaries in the image by analyzing the change rate of pixel grayscale. The well-known edge detection algorithm identifies the edge regions in the image by using the spatial change information or second derivative of the pixel grayscale values in the image. By setting an appropriate threshold, the edge regions can be separated from the background, thereby generating a binary edge image and extracting a clear contour in the image.

[0067] Taking the Canny edge detection algorithm as an example to illustrate the specific implementation method, the Canny edge detection algorithm first smooths the image through a Gaussian filter, effectively reducing the influence of noise and avoiding the interference of noise on the edge detection result. Secondly, by calculating the gradient magnitude and direction, it accurately detects the edge regions with the most significant grayscale changes in the image and can identify small and clear edges. Then, non-maximum suppression is used to ensure the accuracy of edge localization. By retaining the pixel points with the local maximum gradient, the redundant edge responses are removed, making the edge localization more accurate. Finally, a double-threshold detection and edge connection strategy is adopted, which can effectively distinguish strong edges and weak edges and connect the broken edges through the hysteresis effect, making the detected edges continuous and complete.

[0068] Therefore, refer to Figure 3 , first perform grayscale processing on the obtained image containing the hatch to obtain a grayscale image, reduce the dimension of the high-dimensional color information, and convert it into one-dimensional pixel point intensity information to reduce the amount of computation and improve the image processing efficiency; secondly, process the grayscale image through the Canny edge detection algorithm to obtain edges; thirdly, since the painting of the aircraft hatch may cause the edges extracted by the Canny edge detection algorithm to include other edges that do not represent the hatch, it is necessary to screen the detected edges and extract the edges that conform to the shape of the hatch, that is, obtain the contour of the hatch; then, based on the fact that the contour of the hatch is approximately rectangular, the minimum bounding rectangle of the hatch contour is calculated to represent the outer contour of the hatch; finally, after obtaining the minimum bounding rectangle of the hatch contour, judge the pose of the minimum bounding rectangle according to the aspect ratio of the minimum bounding rectangle, and obtain the direction of the minimum bounding rectangle (the aspect ratio greater than 1 is the positive direction), refer to Figure 4, when the minimum bounding rectangle is positive, select the two lower corner points as the lower left corner point and the lower right corner point of the hatch position information. Since the pixel coordinates of the camera are usually in a positive coordinate system, where the origin of the coordinates is at the upper left corner of the image, the horizontal axis is the x-axis, and the vertical axis is the y-axis, sort the four vertices of the minimum bounding rectangle in descending order of the y coordinate to obtain the two points with larger y coordinates as the two lower corner points, and then sort the two lower corner points in ascending order of the x coordinate to obtain the lower left corner point and the lower right corner point respectively, and then obtain that the pixel coordinates of the lower left corner point are D 1L (u 1 l ,v 1 l ), and the pixel coordinates D 1R (u 1 r ,v 1 r ) of the lower right corner point.

[0069] D 1L (u 1 l ,v 1 l ) and D 1R (u 1 r ,v 1 r ) are the position information of the hatch. One or both of the lower left corner point and the lower right corner point can be used as the docking target point for subsequent controlling the actions of the boarding bridge to achieve docking with the hatch.

[0070] Step 2, obtain the position information of the hatch through the YOLOv5 detection algorithm.

[0071] The conventional YOLOv5 object detection model mainly includes an input module, a backbone feature extraction module, a neck feature fusion module, and a prediction output module. The CSP layer in the backbone feature extraction module introduces a partial residual structure, which can maintain the gradient flow of the network and reduce the number of parameters, thereby improving the detection speed. The FPN layer in the neck feature fusion module can fuse features of different resolutions to ensure the detection accuracy of the aircraft hatch.

[0072] To improve the algorithm's ability to extract features of the aircraft cabin door, a lightweight CBAM (Convolutional Block Attention Module) attention mechanism module is added to the backbone feature extraction module. CBAM is an attention mechanism module used to enhance the performance of convolutional neural networks (CNNs). This module can improve performance without increasing the network complexity and enhance the algorithm's ability to extract features of the aircraft cabin door. The module consists of two key parts, namely the channel attention module (C-channel) and the spatial attention module (S-channel).

[0073] The channel attention module performs global max pooling and average pooling operations on the input feature map; and inputs the pooled feature vectors into a shared fully connected layer to learn the attention weights of each channel; then applies the Sigmoid activation function to apply the weights to each channel of the original feature map; finally, the channel feature map is obtained through attention weighting to enhance the feature expression of each channel. The calculation method of the channel attention module is as follows:

[0074]

[0075] Among them, M C is the channel attention map; F is the input feature map; σ is the sigmoid function; AvgPool is global average pooling; MaxPool is global max pooling; MLP is a two-layer neural network, and W 0 , W 1 are the shared weights of the two MLPs; is the average merged feature; is the max merged feature.

[0076] The spatial attention module performs max pooling and average pooling operations on the input feature map along the channel dimension respectively to generate features with different context scales; after concatenating the pooled features along the channel dimension, a feature map with context information of different scales can be generated, and then this feature map is processed by a convolutional layer to generate spatial attention weights; similar to the channel attention mechanism, the Sigmoid activation function is used and the features at each spatial position are weighted to highlight important image regions and reduce unimportant regions. The calculation method of the spatial attention module is as follows:

[0077]

[0078] Among them, M S is the spatial attention map; f 7×7 is a convolutional operation with a filter size of 7×7.

[0079] The CBAM multiplies the output features of the channel attention module and the spatial attention module element-wise to obtain the final attention-enhanced features. The enhanced features will be used as the input for the subsequent network layers to suppress noise and irrelevant information while retaining key information. The diagram of the CBAM module is shown in Figure 5 as follows.

[0080] By improving the conventional YOLOv5 object detection model and adding the above CBAM module to CSP1_3 of the backbone feature extraction, the network can more efficiently obtain the effective information of the aircraft cabin door.

[0081] In addition, in order to achieve the specialized recognition and detection of the aircraft cabin door by the improved YOLOv5 object detection model, images under different lighting conditions and different angles are respectively used when constructing the training dataset of the model, and sufficient data covers various situations to improve the robustness of the network.

[0082] Input the image containing the cabin door collected by the binocular camera into the improved YOLOv5 object detection model. The improved YOLOv5 object detection model outputs the detection result, identifying the cabin door. In the detection result image, the cabin door is surrounded by a detection frame. The coordinate origin of the coordinate system where the detection result image is located is at the upper left corner of the detection result image, the horizontal axis is the x-axis, and the vertical axis is the y-axis. Sort the four vertices of the detection frame according to the y coordinate from large to small to obtain two points with larger y coordinates as the two lower corner points, and then sort the two lower corner points according to the x coordinate from small to large to obtain the lower left corner point and the lower right corner point respectively. Furthermore, the pixel coordinates of the lower left corner point are D 2L (u 2 l , v 2 l ), and the pixel coordinates of the lower right corner point are D 2R (u 2 r , v 2 r ).

[0083] D 2L (u 2 l , v 2 l ) and D 2R (u 2 r , v 2 r ) are the position information of the cabin door. One or both of the lower left corner point and the lower right corner point can be used as the docking target point for the subsequent control of the boarding bridge action to achieve docking with the cabin door.

[0084] Step 3, obtain the optimized cabin door position information through the fuzzy fusion algorithm.

[0085] Since the robustness of the existing technology in detecting the position of the hatch under different working conditions through a single algorithm (based on image processing or deep learning object detection model) is not good, the present invention innovatively uses the fuzzy logic reasoning method to fuse the two detection algorithms to improve the robustness, improve the accuracy and stability of the hatch position detection under different working conditions, and enhance the environmental adaptability.

[0086] Reference Figure 7 , fuzzy logic reasoning is a dynamic model that uses two types of information, data and language, to convert precise numerical data into fuzzy language information according to fuzzy rules, mainly solving complex reasoning problems with fuzzy phenomena. The working mechanism of the fusion model based on fuzzy logic reasoning is as follows: First, the input precise quantity is fuzzified through the fuzzification module and converted into a fuzzy set on the given universe of discourse; secondly, the corresponding fuzzy rules in the rule base are activated, and an appropriate fuzzy reasoning method is selected to obtain the reasoning result according to the known fuzzy facts. Finally, the fuzzy result is defuzzified to obtain the final precise output quantity.

[0087] Step (1), determine the input and output variables of the system.

[0088] According to the actual working conditions of the boarding bridge docking the aircraft hatch, it is determined that the system is a multi-input multi-output system. The four input variables of the system are the state of the boarding bridge (static or dynamic), the ambient light intensity, the distance, and the detection angle. The two output variables of the system are the fusion weight w 1 and the fusion weight w 2 .

[0089] The static state of the boarding bridge refers to the state where the boarding bridge is stationary on the ground, that is, the state when the lifting and walking assembly 10 does not move. The entire docking process starts from the pre-docking point, which is a position at a certain distance from the aircraft hatch at the front end 1 of the receiving port. The dynamic definition is the docking process of the boarding bridge, that is, the process of moving along a preset trajectory from the front end 1 of the receiving port, and the lifting and walking assembly 10 moves; when the front end 1 of the receiving port is at the pre-docking point, it is stationary (at this time, the lifting and walking assembly 10 does not move). From the moment when the front end 1 of the receiving port is at the pre-docking point to the moment when the front end 1 of the receiving port is docked to the hatch, the stationary state of the front end 1 of the receiving port is defined as static.

[0090] As Figure 6 shown, the distance refers to the distance between the detection device and the aircraft hatch (for example, the distances detected by the first distance measuring sensor 4 and the second distance measuring sensor 5, which is actually the distance between the hatch and the front end 1 of the receiving port.

[0091] The ambient light intensity is obtained through a light sensor installed outside the front end of the receiving port.

[0092] The angle sensor measures the angle between the outer channel 9 and the front end 1 of the access port. The angle sensor can be set on the front end 1 of the access port. When the angle value measured by the angle sensor is within [-3°, 3°], it belongs to the frontal view, and when it is within (-45°, -3°) ∪ (3°, 45°), it belongs to the oblique view.

[0093] Step (2), the fuzzification of input and output variables.

[0094] The essence of fuzzification is to convert the given precise numerical data input into a fuzzy set. When the precise value enters the fuzzy inference system, it generally needs to be fuzzified into a fuzzy set on the given universe of discourse. First, establish the fuzzy sets of input and output variables. The set usually consists of the universe of discourse and the membership function. Each element in the universe of discourse is mapped to a membership degree through the membership function, thus forming a set. Among them, the universe of discourse defines the value range of the fuzzy variable, and the membership function describes the membership degree of each element in the universe of discourse.

[0095] Combined with the actual working conditions of the docking operation, fuzzify the variables of this system, that is, divide the corresponding universe of discourse into several sub-intervals and determine the fuzzy levels. The fuzzy levels of the boarding bridge state are divided into: static and dynamic; the fuzzy levels of the light intensity are divided into: strong light, natural light, and weak light; the fuzzy levels of the distance are divided into: close distance, medium distance, and far distance; the fuzzy levels of the viewing angle are divided into: frontal view and oblique view. The fuzzy set of the fusion weight w 1 has the same fuzzy level division as the fusion weight w 2 and both include three parameters: low weight, medium weight, and high weight.

[0096] According to the resolution of the corresponding sensor readings, establish the membership functions applicable to each variable respectively. In this system, triangular, trapezoidal, and Gaussian membership functions are selected respectively. The triangular membership function is simple and intuitive, and has good real-time calculation performance. Its calculation formula is:

[0097]

[0098] where, μ A (x) represents the membership degree corresponding to the input variable x, a is the left base, b is the vertex (the point with a membership degree of 1), and c is the right base.

[0099] The trapezoidal membership function has good robustness to uncertain data and noise, and can stably process fuzzy input data. The expression of the trapezoidal membership function is:

[0100]

[0101] Among them, a, b, c, and d are the four parameters of the trapezoidal membership function, satisfying a ≤ b ≤ c ≤ d. When x ≤ a or x ≥ d, the membership degree μ A (x) = 0, indicating that x does not belong to the current fuzzy set. When a < x ≤ b, the membership degree μ A (x) increases linearly. When b < x ≤ c, the membership degree μ A (x) = 1, indicating that x completely belongs to the current fuzzy set. When c < x ≤ d, the membership degree μ A (x) decreases linearly.

[0102] The Gaussian function is widely adopted in variable fuzzification due to its continuous and smooth curve shape, and its expression is:

[0103]

[0104] Among them, c is the center (mean value) of the membership function, and σ is the standard deviation of the membership function, which controls the width of the membership function.

[0105] In this system, the membership functions of the boarding bridge state and the viewing angle are both set as trapezoidal membership functions; the membership functions of the light intensity and the distance are set as Gaussian functions, and the weights of the two algorithms of the output variable are set as triangular membership functions. The basis for parameter setting refers to the actual working conditions of the boarding bridge docking operation.

[0106] Step (3), establish a fuzzy rule base.

[0107] In a fuzzy inference system, several fuzzy rules form a fuzzy rule base, which forms the core of fuzzy inference. Fuzzy rules generally adopt the "if - then" form. For a given universe of discourse, the n - dimensional fuzzy rule can be expressed as follows:

[0108]

[0109] Among them, R is the decision rule formulated based on manual experience, is the fuzzy set on the universe of discourse of the input variable, and B is the fuzzy set on the universe of discourse of the output variable. The fuzzy rule base of this system is set according to the performance of the two detection processes in the previous step 1 and step 2 under different working conditions, and the weights are allocated according to their performance capabilities. For example, it is known that the detection effect of the detection process in step 1 is better than that of the detection process in step 2 under static, frontal view, weak light, and short - distance conditions. Then the rule setting under this working condition is:

[0110] R 1 : if(x 1 is static)and(x 2 is weak)and(x3 is close)and(x 4 is oblique),then(y 1 is high)、(y 2 is low)

[0111] Step (4), fusion processing.

[0112] Based on the calculated fusion weights w 1 and w 2 , perform fusion processing on the hatch position information obtained in step 1 and the hatch position information obtained in step 2 to obtain the optimized hatch position information F L 、F R . F L The calculation formula of F is as follows:

[0113] F L = w 1 ·D 1L + w 2 ·D 2L

[0114] Where, D 1L is the pixel coordinate of the lower left corner point obtained in step 1, and D 2L is the pixel coordinate of the lower left corner point obtained in step 2.

[0115] F R The calculation formula of F is as follows:

[0116] F R = w 1 ·D 1R + w 2 ·D 2R

[0117] Where, D 1R is the pixel coordinate of the lower right corner point obtained in step 1, and D 2R is the pixel coordinate of the lower right corner point obtained in step 2.

[0118] One or both of the optimized hatch position information F L 、F R are applied as parameters to the subsequent algorithm process for controlling the actions of the boarding bridge. That is, one or both of the optimized lower left corner point and lower right corner point are used as the docking target points for the subsequent control of the boarding bridge actions to achieve docking with the hatch.

[0119] It can be seen that the fuzzy system can dynamically adjust the weights of the two algorithms in step 1 and step 2 according to the actual working conditions, and integrate the results of the two algorithms to obtain the final detection result. The fusion method based on fuzzy logic reasoning improves the flexibility and adaptability of the system, can better cope with complex and changing environments, and improves the accuracy and stability of door position detection under different working conditions.

[0120] For example, 1 is 0.832304, w 2 The x-coordinate value of the target point detected based on image processing is 0.167696, the x-coordinate value of the target point detected based on deep learning is 731.4900, and the x-coordinate value of the target point detected based on deep learning is 723.666991. The fused result can be calculated to be 730.178123.

[0121] The third step is to control the movement of the boarding bridge so that the front end of the pick-up gate docks with the aircraft door.

[0122] Step (a), according to the optimized pixel coordinates of the lower left corner point F L And the pixel coordinates of the lower right corner after optimization F R Calculate the corresponding three-dimensional coordinates in space.

[0123] Combined with the internal parameters of the binocular camera, the pixel coordinates of the lower left corner after optimization are F L And the pixel coordinates of the lower right corner after optimization F R Convert it to the camera coordinate system, and combine it with the depth information obtained based on the binocular camera ranging principle to finally obtain the spatial three-dimensional information F in the camera coordinate system. L (X L ,Y L ,Z L ) and F R (X R ,Y R ,Z R ). For example, the spatial three-dimensional information F corresponding to the lower left corner point L (X L ,Y L ,Z L ) is calculated by the following formula:

[0124]

[0125] The binocular camera has a left camera and a right camera. x is the focal length of the left camera in the X-axis direction, f y is the focal length of the left camera in the Y-axis direction, C x and C y Z is the position of the principal point coordinates of the left camera on the image plane. c is the parallax of the binocular camera, Z cCalculated by the following formula:

[0126]

[0127] where f is the focal length of the binocular camera, B is the horizontal distance between the left camera and the right camera, and x L and x R are the X coordinates of the same three-dimensional target point on the image planes of the left and right cameras.

[0128] In step (2), a well-known control method in the prior art can be used to control the operation of the boarding bridge to complete the docking of the front end of the boarding gate and the cabin door. This well-known control method uses the three-dimensional spatial information F L (X L , Y L , Z L ) and F R (X R , Y R , Z R ) either one or both.

[0129] Or the innovative control method of the present invention can also be used. The innovative control method of the present invention is as follows:

[0130] Combined with the motion model of the boarding bridge, the current position of the front end 1 of the boarding gate, and the docking target point, the motion stroke for the boarding bridge to complete the docking operation can be planned, and a motion command can be generated to control the action of the actuator of the boarding bridge to complete the docking operation. Referring to Figure 9 , the front end 1 of the boarding gate is provided with a floor 1-1, and three rubber cylinders 11 are respectively connected to the front end of the floor 1-1. The rubber cylinders 11 are used to buffer when contacting the fuselage of the aircraft. The central axis 1-1-1 of the floor 1-1 serves as an important reference line during the docking process (the left and right parts of the floor are symmetric about the central axis). The current position of the front end 1 of the boarding gate refers to a docking point selected at the front edge of the floor 1-1, and this docking point is within 10 cm to the left of the central axis 1-1-1.

[0131] Step 1), establish the motion model of the boarding bridge.

[0132] Referring to Figure 8, the main structure of the boarding bridge includes a turntable column 6, a turntable 7, an inner passage 8, an outer passage 9, a lifting and walking assembly 10, and a front end 1 of the passenger boarding door. A horizontal rotation driving mechanism for the front end of the passenger boarding door is also provided between the outer passage 9 and the front end 1 of the passenger boarding door. The horizontal rotation driving mechanism for the front end of the passenger boarding door is used to drive the front end 1 of the passenger boarding door to rotate by a certain angle on the horizontal plane. The power source of the horizontal rotation driving mechanism for the front end of the passenger boarding door is usually a motor. The lifting and walking assembly 10 is provided with differential wheels for controlling the forward, backward, and turning of the boarding bridge; the lifting and walking assembly 10 is also provided with a lifting mechanism. The actuators for controlling the boarding bridge to operate and realizing the docking of the front end 1 of the passenger boarding door with the aircraft cabin door include the lifting and walking assembly 10 and the horizontal rotation driving mechanism for the front end of the passenger boarding door. The control variables for the operation of the lifting and walking assembly 10 are the travel distance of the differential wheels, the rotation angle of the differential wheels, and the lifting height of the lifting mechanism. The control variable for the operation of the horizontal rotation driving mechanism for the front end of the passenger boarding door is the horizontal rotation angle.

[0133] Taking the turntable column 6 as the base coordinate system (X 0 , Y 0 , Z 0 ), the coordinate system where the center of the turntable 7 is located is (X 1 , Y 1 , Z 2 ), the coordinate system where the connection between the inner passage 8 and the turntable 7 is located is (X 2 , Y 2 , Z 2 ), the coordinate system where the center of the differential wheels in the lifting and walking assembly 10 is located is (Z w , Y w , Z w ), and the coordinate system where the center point of the floor of the front end 1 of the passenger boarding door is located is (X c , Y c , Z c ).

[0134] The transformation relationship from the base coordinate system to the center point of the floor of the front end 1 of the passenger boarding door and the transformation relationship from the base coordinate system to the center of the differential wheels in the lifting and walking assembly 10 are obtained respectively through forward kinematics and the transformation relationship from the base coordinate system to the center of the differential wheels in the lifting and walking assembly 10 According to the installation position of the binocular camera, the external parameters from the camera coordinate system to the center point of the floor of the front end 1 of the passenger boarding door can be obtained The transformation relationship from the base coordinate system to the camera coordinate system

[0135] Furthermore, the motion model of the boarding bridge is established.

[0136] Step 2), based on the current value of the travel distance of the differential wheels of the lifting and walking assembly, the current value of the rotation angle of the differential wheels, the current value of the lifting height of the lifting mechanism, the current value of the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the boarding bridge, the three-dimensional coordinates of the docking target point, and the current position information of the front end 1 of the boarding bridge, reverse solve the target value of the travel distance of the differential wheels, the target value of the rotation angle of the differential wheels, the target value of the lifting height of the lifting mechanism, and the target value of the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the boarding bridge through the motion model of the boarding bridge based on the inverse kinematics principle.

[0137] When the boarding bridge is in the starting position, the above transformation relationships can be used to calculate respectively Using the three-dimensional coordinates of the docking target point and the current position, a set of equations with control parameter variables can be obtained, and by solving the equations, the motion stroke of the boarding bridge to achieve docking with the aircraft cabin door can be calculated.

[0138] Combined with the motion mechanism of the boarding bridge, to achieve the docking operation, it is necessary to complete the forward stroke, angle adjustment of the lifting and walking assembly 10, the lifting of the boarding bridge, and the rotation of the horizontal rotation drive mechanism at the front end of the boarding bridge. Among them, the docking point on the floor of the front end 1 of the boarding bridge is used as the starting point of the current position, and the detected lower left corner of the cabin door is used as the docking target point (i.e., the three-dimensional information F in space L (X L , Y L , Z L ) represents the docking target point), and the motion trajectory of the boarding bridge when moving from the starting point to the docking target point can be calculated. Then, a motion instruction is generated and sent to the actuator of the boarding bridge, and the actuator acts to control the docking operation. The motion instructions here include: controlling the travel distance of the differential wheels in the lifting and walking assembly 10, the rotation angle of the differential wheels, controlling the lifting height of the lifting mechanism in the lifting and walking assembly 10, and controlling the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the boarding bridge.

[0139] With the help of the first distance measuring sensor 4 and the second distance measuring sensor 5, the angle that the front end of the boarding bridge needs to adjust can be calculated. The principle is as follows: two vectors can be obtained respectively according to the installation positions of the left and right distance measurements and the detected lower left and right corner points, and by calculating the included angle between the two vectors, the rotation angle of the front end of the boarding bridge can be determined.

[0140] It should be noted that when using the lower right corner point as the docking target point, the docking point selected at the front edge of the floor 1-1 of the front end 1 of the boarding bridge is located on the right side of the central axis 1-1-1 and close to the right end point of the front edge of the floor 1-1, for reference Figure 10 .

[0141] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the present invention.

Claims

1. A method for detecting aircraft door posture based on fuzzy fusion, characterized in that: The following steps are involved: Step 1, obtaining the position information of the hatch through an image processing algorithm; For the image containing the cabin door captured by the binocular camera, the edge detection algorithm is used to obtain the outline of the cabin door. First, the grayscale image containing the cabin door is processed to obtain a grayscale image, and the high-dimensional color information is reduced and converted into one-dimensional pixel intensity information; secondly, the grayscale image is processed by the Canny edge detection algorithm to obtain the edge; Secondly, extract the edge that matches the shape of the hatch, that is, get the outline of the hatch; The minimum circumscribed rectangle of the hatch outline is obtained to represent the outer contour of the hatch; The posture of the minimum enclosing rectangle is determined according to the aspect ratio of the minimum enclosing rectangle, and the direction of the minimum enclosing rectangle is obtained. The aspect ratio is greater than 1, which is positive. When the minimum enclosing rectangle is positive, the four vertices of the minimum enclosing rectangle are sorted from large to small according to the y coordinates, and the two points with larger y coordinates are obtained as the two corner points at the bottom. Then, the two corner points at the bottom are sorted from small to large according to the x coordinates, and the lower left corner point and the lower right corner point are obtained respectively, and then the pixel coordinates of the lower left corner point are obtained. and / or the pixel coordinates of the lower right corner ; Step 2, obtaining the location information of the door through the YOLOv5 detection algorithm; The YOLOv5 target detection model was established through the training data set. When constructing the training data set, images under different lighting conditions and angles were taken. The image containing the cabin door captured by the binocular camera is input into the YOLOv5 target detection model. The YOLOv5 target detection model outputs the detection result and identifies the cabin door. In the detection result image, the cabin door is surrounded by a detection frame. The four vertices of the detection frame are sorted from large to small according to the y coordinate, and the two points with larger y coordinates are obtained as the two corner points at the bottom. Then the two corner points at the bottom are sorted from small to large according to the x coordinate, and the lower left corner point and the lower right corner point are obtained respectively. Then the pixel coordinates of the lower left corner point are obtained as follows: and or the pixel coordinates of the lower right corner ; Step 3, obtaining the optimized door position information through fuzzy fusion algorithm; Step (1), determine the system input and output variables; The system's multiple input variables are the state of the boarding bridge, the ambient light intensity, the distance, and the detection angle of view; the fuzzy level of the boarding bridge state is divided into: static and dynamic; the fuzzy level of the light intensity is divided into: strong light, natural light, and weak light; the fuzzy level of the distance is divided into: close distance, medium distance, and long distance; the fuzzy level of the angle of view is divided into: normal view and oblique view; the fusion weight Fuzzy Sets and Fusion Weights The fuzzy classification of is the same, including three parameters: low weight, medium weight and high weight; The two output variables of the system are fusion weights and fusion weight ; Step (2), fuzzification of input and output variables; Step (3), establish a fuzzy rule base; Step (4), fusion processing; According to the fusion weight and , the door position information obtained in step 1 and the door position information obtained in step 2 are fused to obtain the optimized door position information and / or , The calculation formula is as follows: in, is the pixel coordinate of the lower left corner point obtained in step 1, is the pixel coordinate of the lower left corner point obtained in step 2; The calculation formula is as follows: in, is the pixel coordinate of the lower right corner point obtained in step 1, is the pixel coordinate of the lower right corner point obtained in step 2; Step 4: Based on the optimized door position information and / or , combined with the depth information obtained based on the binocular camera ranging principle, the spatial three-dimensional information in the camera coordinate system is obtained and / or .

2. The aircraft door posture detection method based on fuzzy fusion according to claim 1 is characterized in that: In the step 2, a CBAM module is added to the YOLOv5 target detection model to obtain an improved YOLOv5 target detection model, and the improved YOLOv5 target detection model is used to perform recognition and detection of the hatch.

3. A method for controlling automatic docking of a boarding bridge, characterized in that: The optimized door position information described in any one of claims 1-2 is used.

4. A method for automatically docking an aerobridge using the optimized rear door position information obtained by the detection method of claim 1, characterized in that: The following steps are involved: Step 1), establish the motion model of the boarding bridge; The boarding bridge comprises a turntable column, a turntable, an inner channel, an outer channel, a lifting and walking assembly and a front end of a receiving port. A horizontal rotation driving mechanism for the front end of the receiving port is also provided between the outer channel and the front end of the receiving port. The lifting and walking assembly is provided with a differential wheel for controlling the boarding bridge to move forward, backward and turn. The lifting and walking assembly is also provided with a lifting mechanism. The control quantity of the lifting and traveling assembly is the travel distance of the differential wheel, the rotation angle of the differential wheel, and the lifting height of the lifting mechanism. The control quantity of the horizontal rotation drive mechanism at the front end of the receiving port is the horizontal rotation angle. Take the turntable column as the base coordinate , the coordinates of the center of the turntable are , the coordinates of the connection between the inner channel and the turntable are , the coordinates of the center of the differential wheel in the lifting and traveling assembly are , the coordinates of the center point of the floor in front of the pick-up entrance are ; Step 2), solve the target value of the control quantity through inverse kinematics; According to the current value of the differential wheel travel distance of the lifting and traveling assembly, the current value of the differential wheel rotation angle, the current value of the lifting height of the lifting mechanism, the current value of the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the receiving gate, the three-dimensional coordinates of the docking target point and the current position information of the front end of the receiving gate, the target value of the differential wheel travel distance, the target value of the differential wheel rotation angle, the target value of the lifting height of the lifting mechanism, and the target value of the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the receiving gate are reversely solved through the motion model of the boarding bridge based on the inverse kinematics principle; Step 3), based on the target value of the differential wheel travel distance, the target value of the differential wheel rotation angle, the target value of the lifting height of the lifting mechanism, and the target value of the horizontal rotation angle of the horizontal rotation drive mechanism at the front end of the docking port, a motion instruction is generated and sent to the corresponding actuator in the boarding bridge, so that the actuator is activated to perform the docking operation.

Images