Apron berth guidance method based on image recognition technology

Abstract

This invention provides an apron parking guidance method based on image recognition technology, belonging to the field of parking guidance technology. The method includes the following steps: responding to a parking signal emitted by the aircraft, acquiring an aircraft parking image, the aircraft's current speed, and its direction of travel; the aircraft parking image includes the aircraft to be parked and the centerline of the apron parking space; identifying the nose wheel light, left wing light, and right wing light of the aircraft to be parked in the parking image, and establishing an aircraft attitude model based on the apron parking space centerline; establishing an aircraft speed model based on the aircraft's current speed and direction of travel; establishing an aircraft parking guidance model based on the aircraft's attitude and speed models; and outputting aircraft speed and direction of travel adjustment data based on the aircraft parking guidance model. This invention effectively solves the problem of low recognition accuracy or algorithm failure in existing related technologies under adverse weather conditions.

Description

Technical Field

[0001] This invention relates to the field of parking guidance technology, and specifically to a method for guiding apron berths based on image recognition technology. Background Technology

[0002] Currently, the parking process for aircraft is generally handled by manual guidance or an automated guidance system. Manual guidance suffers from low efficiency, high labor costs, and lower safety, while automated guidance systems have become the mainstream approach. An automated guidance system (Airport Docking Auto-guide System, ADGS) provides pilots with information such as the distance between the aircraft's current position and the parking position, the aircraft's taxiing speed, and whether the aircraft has deviated from the taxi centerline during the journey from the taxiway to the parking space. This information enables pilots to accurately park the aircraft on the parking space.

[0003] However, existing automatic guidance systems are not very accurate in guiding berths in adverse weather conditions such as low light, rain, snow, and dust. In some cases, the algorithms may fail due to the adverse weather conditions. For example, in image recognition-based automatic guidance schemes, the aircraft's outline may not be recognized due to the influence of adverse weather, which may lead to the ineffectiveness of subsequent guidance algorithms. Summary of the Invention

[0004] To address the technical problems in related technologies, this invention provides a method for guiding apron berths based on image recognition technology.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] Apron berth guidance methods based on image recognition technology include:

[0007] Step S1: In response to the parking signal issued by the aircraft, acquire the aircraft parking image, the aircraft's current speed and direction of travel, wherein the aircraft parking image includes the aircraft to be parked and the center line of the apron berth;

[0008] Step S2: Identify the nose wheel light, left wing light, and right wing light of the aircraft to be parked in the aircraft parking image, and establish the aircraft's attitude model in conjunction with the center line of the apron parking space; establish the aircraft's speed model in conjunction with the aircraft's current speed and direction of travel.

[0009] Step S3: Establish an aircraft parking guidance model based on the aircraft's attitude and velocity models;

[0010] Step S4: Output the aircraft's speed and direction of travel adjustment data based on the aircraft's parking guidance model.

[0011] Optionally, step S1 further includes:

[0012] Step S1-1: Perform spectral normalization on the acquired aircraft parking image according to the following formula:

[0013]

[0014] In the formula, I(x,y,c) is the pixel value of color channel c at coordinates (x,y) of the original image, and μ c Let σ be the global mean of channel c. c Let α be the standard deviation of channel c. c β is the gain coefficient for channel c. c This is the bias term for channel c;

[0015] Step S1-2: Spatial sharpening of the image obtained in step S1-1 is performed according to the following formula:

[0016]

[0017] In the formula, For the Laplace operator, G σ (x,y) is a Gaussian kernel with standard deviation σ, and λ is the sharpening weight coefficient.

[0018] Optionally, step S2 includes:

[0019] Establish the aircraft's attitude model:

[0020] Step S2-1-1: Convert the image obtained in step S1-2 to HSV space and separate the hue (H) and saturation (S) channels:

[0021]

[0022]

[0023] In the formula, R is the normalized value of the red channel, G is the normalized value of the green channel, and B is the normalized value of the blue channel.

[0024] Step S2-1-2: Perform image segmentation based on the color threshold range and saturation threshold range of the nose wheel light, left wing light and right wing light to identify the nose wheel light, left wing light and right wing light of the aircraft to be parked in the aircraft parking image respectively.

[0025] Step S2-1-3: Combining the centerline of the apron berth in the aircraft parking image and the known wingspan parameters of the aircraft, construct the spatial mapping relationship between the aircraft body coordinate system and the coordinate system of the apron berth centerline;

[0026] Step S2-1-4: Use the perspective projection transformation model to transform the two-dimensional image coordinates into three-dimensional spatial coordinates:

[0027]

[0028] In the formula, K is the camera intrinsic parameter matrix, (x,y) are the image pixel coordinates, and d is the depth estimate. L W X represents the wingspan of the aircraft. R Let X be the coordinate of the right wing light on the X-axis. L The coordinates of the left wing light on the X-axis;

[0029] Step S2-1-5: Using the geometric constraints between the spatial triangle formed by the coordinates of the nose wheel light, left wing light, and right wing light and the centerline of the apron berth, calculate the aircraft's pitch angle θ, roll angle Φ, and yaw angle Ψ respectively.

[0030]

[0031] In the formula, (X L ,Y L Z L (X) represents the three-dimensional coordinates of the left wing light. R ,Y R Z R (X) represents the three-dimensional coordinates of the right wing light. nose Z nose (x) represents the coordinates of the nose wheel light. center Let L be the coordinate of the apron berth on the X-axis. L This refers to the fuselage length.

[0032] Optionally, step S2 further includes:

[0033] Establish an aircraft speed model:

[0034] Step S2-2-1: Perform velocity vector decomposition according to the following formula:

[0035]

[0036] In the formula, (X t ,Y t (X) represents the position coordinates of the nose wheel light at time t. t-1 ,Y t-1 ) represents the position coordinates of the nose wheel light at time t-1, Δt is the time interval between consecutive frames, and ω is the yaw rate;

[0037] Step S2-2-2: Calculate the effective taxiing speed and lateral offset speed of the aircraft along the centerline of the apron berth according to the following formula:

[0038]

[0039] Where, V || is the effective taxiing speed of the aircraft along the center line of the apron berth, and V ⊥ is the lateral offset speed of the aircraft.

[0040] Optionally, the step S3 includes:

[0041] Establish a parking guidance model for the aircraft:

[0042] Step S 3-1: Set a comprehensive evaluation function:

[0043]

[0044] Where, α, β, and δ are weight coefficients respectively, and D offset is the offset distance between the nose wheel light and the center line of the apron berth and Y<00MASK>[000026]] nose is the coordinate of the nose wheel light on the Y-axis;

[0045] Step S 3-2: Output a corresponding instruction according to the value of the comprehensive evaluation function:

[0046] When F > 0.8, output a red warning signal;

[0047] When 0.5 < F ≤ 0.8, output a yellow speed limit prompt signal;

[0048] When F ≤ 0.5, output a green passing signal.

[0049] Optionally, the step S4 includes:

[0050] Step S 4-1: Calculate the direction correction angle:

[0051]

[0052] Where, k1 and k2 are PID coefficients, L remaining is the remaining taxiing distance, ΔD is the offset of the aircraft center line, and ΔD = Y nose - Y centerline , Y centerline is the coordinate value of the center line of the apron berth on the Y-axis;

[0053] Step S 4-2: Calculate the recommended taxiing speed:

[0054]

[0055] Step S 4-3: Output the calculated direction correction angle and recommended taxiing speed.

[0056] Optionally, in the step S 2-1-2,

[0057] The color threshold range of the nose wheel light is: H∈[0°,30°]∪[330°,360°];

[0058] The color threshold range of the left wing light is: H∈[0°,15°];

[0059] The color threshold range of the right wing light is: H∈[90°,150°];

[0060] The saturation threshold is: S>0.7.

[0061] Optionally, step S2-1-1 further includes:

[0062] Step S2-1-1-1: Remove isolated noise points using opening operation:

[0063]

[0064] Step S2-1-1-2: Connect the fractured regions using closing operations:

[0065]

[0066] In the formula, For corrosion operation, For the expansion operation, the structural element W is a 3×3 circular core.

[0067] Optionally, step S2-1-2 further includes:

[0068] Based on the aircraft's geometric characteristics, establish spatial positional constraints for the nose wheel light, left wing light, and right wing light:

[0069] For the nose wheel light, its spatial position is located in the lower 1 / 3 area of ​​the aircraft parking image;

[0070] The left and right wing lights are positioned higher than the nose wheel lights and within the left and right quarters of the aircraft's parking image.

[0071] Beneficial effects:

[0072] 1. Through the above technical solution, firstly, since the method of the present invention is based on the nose wheel light, left wing light, right wing light and the center line of the apron berth of the aircraft to be parked to perform attitude recognition and establish an attitude model, even under adverse weather conditions such as dust, low light, rain and snow, these lights can still be effectively and accurately identified. The method of the present invention can effectively ensure the effective recognition of the attitude of the aircraft to be parked, and can effectively solve the problem of low recognition accuracy or recognition algorithm failure of the existing related technologies under adverse weather conditions.

[0073] Secondly, this invention first constructs a three-dimensional attitude model by combining the position of the lights and the centerline of the apron berth, effectively avoiding the sensitivity of traditional two-dimensional contour recognition to complex weather conditions. Then, this invention establishes a velocity model of the aircraft using its current speed and direction of travel, providing a more accurate reference and calculation basis for parking guidance. This method, through dual modeling of the aircraft's attitude model (spatial geometric relationships) and velocity model (kinematic analysis), effectively overcomes the limitations of single sensors, such as the scattering problem of lidar in rain and snow environments, and effectively improves the reliability of guidance.

[0074] Third, the method of the present invention forms a closed-loop control link from image acquisition to output adjustment, reducing manual intervention. Furthermore, the method of the present invention only identifies three light positions and the center line of the apron berth. Compared with existing related technologies for identifying aircraft feature points, related contours and overall images, the identification and processing cycle of the present invention is shorter than that of existing related technologies, and it has stronger real-time performance and speed.

[0075] 2. Other beneficial effects or advantages of the present invention will be described in detail in the specific embodiments. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] in:

[0078] Figure 1 This is a flowchart illustrating the steps of an apron berth guidance method based on image recognition technology provided in an exemplary embodiment of the present invention.

[0079] Figures 2 to 5 This is a schematic diagram of the display content of a parking instruction device provided in an exemplary embodiment of the present invention. The display content of the instruction device includes five parts: the aircraft type currently berthing, the adjustment direction, the remaining taxiing distance, a T-shaped schematic diagram of the parking space, and a schematic diagram of the aircraft position.

[0080] in, Figure 2 The image shows an A320 aircraft facing the centerline of its parking space, with 29 meters of remaining taxiway.

[0081] Figure 3The image shows an A320 aircraft positioned to the left of the centerline of its parking space, with its direction adjusted to the left. The aircraft has 20 meters of remaining taxiway.

[0082] Figure 4 The image shows an A320 aircraft positioned to the left of the centerline of its parking space, with its direction adjusted to the right. The aircraft has 20 meters of remaining taxiway.

[0083] Figure 5 The image shows an A320 aircraft parked in its designated spot. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0085] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0086] To facilitate a clearer and more accurate understanding of the technical solution of this invention by those skilled in the art, the following section will further explain the problems existing in the current automatic guidance system.

[0087] Taking an image recognition-based automatic guidance scheme as an example, it first acquires images of the aircraft to be parked, identifies its aircraft type and current attitude by recognizing its feature points, related contours, and overall image, and then guides it to the parking space. However, in adverse weather conditions (such as sandstorms, rain, snow, low light, etc.), the image recognition process may have difficulty accurately recognizing its feature points and related contours, making it impossible to accurately determine the aircraft's current attitude, which in turn leads to the ineffectiveness of subsequent guidance algorithms.

[0088] Based on this, the present invention provides a novel solution: an apron parking guidance method based on image recognition technology. This method utilizes the aircraft's left wing light (typically a red light at the left wingtip), right wing light (typically a green light at the right wingtip), and nose wheel light (typically a white light for illumination located at the nose landing gear position), combined with the centerline of the parking space, to accurately determine the aircraft's current attitude, thereby enabling accurate parking guidance. These lights can be effectively identified even in adverse weather conditions, thus making it suitable for guidance needs under such conditions.

[0089] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0090] Please see Figure 1 This invention provides a method for guiding apron berths based on image recognition technology, comprising the following steps:

[0091] Step S1: In response to the parking signal issued by the aircraft, acquire the aircraft parking image, the aircraft's current speed and direction of travel. The aircraft parking image includes the aircraft to be parked and the center line of the apron berth.

[0092] Step S2: Identify the nose wheel light, left wing light, and right wing light of the aircraft to be parked in the aircraft parking image, and establish the aircraft's attitude model in conjunction with the center line of the apron parking space; establish the aircraft's speed model in conjunction with the aircraft's current speed and direction of travel.

[0093] Step S3: Establish an aircraft parking guidance model based on the aircraft's attitude and velocity models;

[0094] Step S4: Output the aircraft's speed and direction of travel adjustment data based on the aircraft's parking guidance model.

[0095] Through the above technical solution, firstly, since the method of the present invention is based on the nose wheel light, left wing light, right wing light and the center line of the apron berth of the aircraft to be parked to perform attitude recognition and establish an attitude model, even under adverse weather conditions such as dust, low light, rain and snow, these lights can still be effectively and accurately identified. The method of the present invention can effectively ensure the effective recognition of the attitude of the aircraft to be parked, and can effectively solve the problem of low recognition accuracy or recognition algorithm failure in the existing related technologies under adverse weather conditions.

[0096] Secondly, this invention first constructs a three-dimensional attitude model by combining the position of the lights and the centerline of the apron berth, effectively avoiding the sensitivity of traditional two-dimensional contour recognition to complex weather conditions. Then, this invention establishes a velocity model of the aircraft using its current speed and direction of travel, providing a more accurate reference and calculation basis for parking guidance. This method, through dual modeling of the aircraft's attitude model (spatial geometric relationships) and velocity model (kinematic analysis), effectively overcomes the limitations of single sensors, such as the scattering problem of lidar in rain and snow environments, and effectively improves the reliability of guidance.

[0097] Third, the method of the present invention forms a closed-loop control link from image acquisition to output adjustment, reducing manual intervention. Furthermore, the method of the present invention only identifies three light positions and the center line of the apron berth. Compared with existing related technologies for identifying aircraft feature points, related contours and overall images, the identification and processing cycle of the present invention is shorter than that of existing related technologies, and it has stronger real-time performance and speed.

[0098] In one embodiment of the present invention, step S1 may further include:

[0099] Step S1-1: Perform spectral normalization on the acquired aircraft parking image according to the following formula:

[0100]

[0101] In the formula, I(x,y,c) is the pixel value of color channel c at coordinates (x,y) of the original image, and μ c σ is the global mean of channel c, used to characterize the ambient light intensity. c α is the standard deviation of channel c, used to characterize ambient light fluctuations. c β is the gain coefficient for channel c, used to enhance the target band. c This is the bias term for channel c, used to compensate for weak light attenuation;

[0102] Step S1-2: Spatial sharpening of the image obtained in step S1-1 is performed according to the following formula:

[0103]

[0104] In the formula, The Laplacian operator is used to extract high-frequency edges, G σ (x,y) is a Gaussian kernel with a standard deviation of σ, used to control the smoothing scale, and λ is a sharpening weight coefficient used to balance details and noise.

[0105] In this embodiment, the present invention proposes a two-stage image enhancement technique of spectral normalization processing and spatial sharpening processing, which aims to solve the problem of feature recognition difficulties caused by image quality degradation under adverse weather conditions.

[0106] Specifically, firstly, the spectral normalization process of this invention (step S1-1) can compensate for weak light attenuation (such as low light intensity or dust obstruction) and suppress ambient light fluctuations, thereby improving the contrast of the target wavelength band (such as aircraft lights). For example, in dusty weather, the color of light is attenuated due to scattering, which can be mitigated by adjusting the gain coefficient α. c Enhance the red / green bands (corresponding to the left and right wing lights), offset term β c Compensating for signal loss in low light conditions makes light features easier to segment.

[0107] Secondly, the spatial sharpening process (steps S1-2) of this invention strengthens high-frequency edges (such as light contours) using a Laplacian-Gaussian mixture operator, while smoothing background noise (such as rain and snow particles) using a Gaussian kernel, thereby effectively reducing the false recognition rate. For example, in a rainy day image, raindrops form random noise, and the Gaussian kernel G... σ Smooth background, Laplace operator The sharp edges of the light are extracted, and the sharpening weight coefficient λ controls the sharpening intensity to avoid noise amplification.

[0108] Third, based on the global mean μ c and standard deviation σ c The adaptive normalization can eliminate the impact of changes in ambient light intensity (e.g., sudden changes in illumination caused by cloud cover) on the image, ensuring the stability of feature segmentation. For example, in low-light environments, μ... c Characterizing the overall ambient light intensity, σ c By reflecting light fluctuations and normalizing the image brightness to be consistent across different time periods, the algorithm can be effectively prevented from failing due to changes in lighting.

[0109] In this embodiment, it should be noted that, firstly, for the spectral normalization formula, the global mean μ c and standard deviation σ c Used to calculate global statistics for each channel to eliminate differences in ambient lighting (e.g., color temperature differences between cloudy and sunny days). Gain coefficient α c Used to enhance the target band (e.g., the R-channel gain corresponding to the red left wing light) and suppress irrelevant bands (e.g., the yellow band against a dusty background). Bias term β c It is used to compensate for nonlinear attenuation under low light conditions (e.g., insufficient light signal strength at night) and improve visibility in low signal-to-noise ratio areas.

[0110] Second, for the spatial sharpening formula, the Laplacian operator Used to extract the second derivative of the image (high-frequency edges) to enhance the transition areas between light and background. Gaussian kernel G σ Used to smooth low-frequency noise (e.g., the graininess of rain, snow, and dust), controlling the scale range of sharpening. The sharpening weight coefficient λ is used to balance detail enhancement and noise suppression; for example, λ can be decreased in rainy or snowy weather to reduce high-frequency interference, while λ can be increased in dusty weather to enhance blurred contours.

[0111] Third, firstly, compared to traditional histogram equalization methods, which often lead to overexposure due to global brightness adjustments, the method of this invention preserves color distribution characteristics through channel normalization, thus avoiding color distortion of the lights. Secondly, compared to traditional single sharpening algorithms, which tend to amplify noise, the method of this invention combines Gaussian smoothing (to suppress noise) with weighted fusion (sharpening weight coefficient λ), achieving a balance between detail and noise.

[0112] In one embodiment of the present invention, step S2 may include:

[0113] Establish the aircraft's attitude model:

[0114] Step S2-1-1: Convert the image obtained in step S1-2 to HSV space and separate the hue (H) and saturation (S) channels:

[0115]

[0116] In the formula, R is the normalized value of the red channel, i.e., I norm (x,y,R), where G is the normalized value of the green channel, i.e., I norm (x, y, G), where B is the normalized value of the blue channel, i.e., I norm (x,y,B);

[0117] Step S2-1-2: Perform image segmentation based on the color threshold range and saturation threshold range of the nose wheel light, left wing light and right wing light to identify the nose wheel light, left wing light and right wing light of the aircraft to be parked in the aircraft parking image respectively.

[0118] Step S2-1-3: Combining the centerline of the apron berth in the aircraft parking image and the known wingspan parameters of the aircraft, construct the spatial mapping relationship between the aircraft body coordinate system and the coordinate system of the apron berth centerline;

[0119] Step S2-1-4: Use the perspective projection transformation model to transform the two-dimensional image coordinates into three-dimensional spatial coordinates:

[0120]

[0121] In the formula, K is the camera intrinsic parameter matrix, (x,y) are the image pixel coordinates, and d is the depth estimate. L W X represents the wingspan of the aircraft. R Let X be the coordinate of the right wing light on the X-axis. L The coordinates of the left wing light on the X-axis;

[0122] Step S2-1-5: Using the geometric constraints between the spatial triangle formed by the coordinates of the nose wheel light, left wing light, and right wing light and the centerline of the apron berth, calculate the aircraft's pitch angle θ, roll angle Φ, and yaw angle Ψ respectively.

[0123]

[0124] In the formula, (X L ,Y L Z L (X) represents the three-dimensional coordinates of the left wing light. R ,Y R Z R (X) represents the three-dimensional coordinates of the right wing light. nose Z nose (x) represents the coordinates of the nose wheel light. center Let L be the coordinate of the apron berth on the X-axis. L This refers to the fuselage length.

[0125] In this embodiment, the aircraft attitude modeling method based on HSV color space segmentation and three-dimensional geometric constraints solves the problems of inaccurate light feature segmentation and large attitude calculation errors under adverse weather conditions. Specifically, firstly, the method of this invention separates the hue (H) and saturation (S) channels in the HSV space, and combines the color threshold and saturation threshold (e.g., S>0.7 as described below) to effectively distinguish light from background noise (e.g., rain and snow reflections, dust spots, etc.). For example, in dusty weather, by combining the H value range of the left wing light (red) and the right wing light (green) with the saturation threshold, low-saturation interference points (e.g., dust reflections) can be effectively filtered out.

[0126] Second, the method of the present invention is based on known wingspan parameters (L W The coordinate system of the apron berth centerline is transformed through perspective projection. Mapping two-dimensional image coordinates to three-dimensional space can effectively eliminate the impact of perspective distortion on attitude calculation. For example, by mapping the distance between the left and right wing lights (X... R -X L ) Inversely deduce the true depth Millimeter-level positioning error is achieved by combining the camera intrinsic parameter matrix (K).

[0127] Third, the method of this invention utilizes the geometric relationship between the spatial triangle formed by the nose wheel light, left and right wing lights, and the apron centerline (such as the formulas for pitch angle θ, roll angle Φ, and yaw angle Ψ) to achieve joint attitude calculation across multiple degrees of freedom, effectively reducing attitude errors. Specifically, the yaw angle Ψ is calculated using the coordinate difference (Y) between the left and right wing lights. R -Y L The calculation can eliminate the interference of aircraft lateral deviation on attitude estimation.

[0128] In this embodiment, it should be noted that, firstly, regarding the HSV space partitioning formula of the present invention ( and In this context, H (hue) is calculated by the difference between the RGB channels to separate the light color (such as red and green) from the background color (such as yellow sand). S (saturation) is used to filter out non-light source areas (such as low-saturation reflections) and retain high-saturation targets.

[0129] Second, regarding the three-dimensional coordinate mapping formula of this invention In this context, K (camera intrinsic matrix) is used to correct lens distortion and ensure a linear mapping between pixel coordinates (x,y) and physical space coordinates (X,Y,Z). d (depth estimate) is used to infer the true distance using the known wingspan and the distance between the left and right wing lights, avoiding the matching error of traditional binocular vision.

[0130] Third, the formula for calculating the pitch angle θ is based on the offset between the nose wheel light and the apron centerline (X). center -X nose ) and fuselage length (L L The calculation reflects the aircraft's longitudinal roll. The formula for calculating the roll angle Φ is based on the nose wheel light height (Z). nose ) and the height difference between the wing lights (Z) L -Z R The calculation reflects the aircraft's lateral balance. The formula for calculating the yaw angle Ψ is based on the difference in coordinates between the left and right wing lights (Y). R -Y L The calculations reflect the degree to which the aircraft's direction of travel deviates from the centerline.

[0131] Fourth, compared with existing RGB threshold segmentation methods, which are easily affected by light intensity, the method of this invention improves the stability of color segmentation by separating hue and saturation. Furthermore, compared with traditional single-sensor positioning methods, traditional LiDAR or IMUs are highly dependent on hardware precision, while the method of this invention achieves low-cost, high-precision, and efficient computation through image geometric constraints.

[0132] In one embodiment of the present invention, step S2 may further include:

[0133] Establish an aircraft speed model:

[0134] Step S2-2-1: Perform velocity vector decomposition according to the following formula:

[0135]

[0136] In the formula, (X t ,Y t (X) represents the position coordinates of the nose wheel light at time t. t-1 ,Y t-1 ) represents the position coordinates of the nose wheel light at time t-1, Δt is the time interval between consecutive frames, and ω is the yaw rate;

[0137] Step S2-2-2: Calculate the effective taxiing speed and lateral offset speed of the aircraft along the centerline of the apron berth according to the following formula:

[0138]

[0139] In the formula, V || V is the effective taxiing speed of the aircraft along the centerline of the apron berth. ⊥ This represents the aircraft's lateral drift velocity.

[0140] In this embodiment, firstly, traditional image recognition may fail due to blurred contours, while the method of the present invention calculates the velocity component (V) by tracking stable luminous feature points such as nose wheel lights (rather than relying on the overall contour) and utilizing positional changes between consecutive frames. x V y By using ω), the instantaneous motion state of the aircraft can be obtained in real time. This feature point tracking method can effectively enhance the robustness in low light, rain and snow scenarios.

[0141] Second, the method of the present invention decomposes the velocity in the global coordinate system into an effective taxiing velocity (V) along the centerline of the berth. || ) and lateral offset velocity (V ⊥ This allows for a clear distinction between the two motion trends of the aircraft: V || Used to reflect the forward taxiing efficiency of an aircraft toward a parking space, in order to determine whether acceleration or deceleration is necessary. V ⊥ This is used to indicate the lateral deviation trend of the aircraft from the centerline, so as to trigger a direction correction command. This decomposition method allows the guidance algorithm to adjust speed and direction in a targeted manner, avoiding blind operation.

[0142] Third, in the speed model of the present invention, the yaw angle (Ψ) is introduced as a coordinate transformation parameter to project the global speed into a local coordinate system based on the center line of the berth. In this way, the speed calculation can follow the change of the aircraft attitude in real time, avoiding misjudgment of the speed caused by the aircraft turning, and thus ensuring that the model output is consistent with the current actual motion state.

[0143] In this embodiment, it should be noted that, first, for the speed vector decomposition formula, the instantaneous linear velocity (V t , V t ) is calculated through the position change (X x , Y y ) of the nose wheel light in consecutive frames, and the turning trend is captured in combination with the yaw angle change rate (ω). This method is applied to the tracking of specific feature points, avoiding the dependence on overall contour recognition in the existing related technologies.

[0144] Second, for the calculation formulas of the effective taxiing speed and the lateral offset speed, the speed in the global coordinate system is projected into a local coordinate system based on the center line of the berth by using the yaw angle (Ψ). Among them, V<00?00093> represents the actual taxiing speed of the aircraft along the center line direction, which is used to evaluate whether it is necessary to adjust the taxiing efficiency. V ⊥ represents the lateral offset speed, reflecting the trend of the aircraft deviating from the center line, which is used to trigger direction correction.

[0145] In an embodiment of the present invention, step S3 of the present invention may include:

[0146] Establish a parking guidance model for the aircraft:

[0147] Step S3-1: Set a comprehensive evaluation function:

[0148]

[0149] In the formula, α, β, and δ are weight coefficients respectively, D offset is the offset distance between the nose wheel light and the center line of the apron berth and Y nose is the coordinate of the nose wheel light on the Y-axis;

[0150] Step S3-2: Output corresponding instruction commands according to the value of the comprehensive evaluation function:

[0151] When F > 0.8, output a red warning signal;

[0152] ]>When 0.5 < F ≤ 0.8, output a yellow speed limit prompt signal;

[0153] When F ≤ 0.5, output a green passing signal.

[0154] In this embodiment, first, the present invention constructs a comprehensive evaluation function F by integrating three key parameters: the attitude angle deviation (|Ψ|) of the aircraft, the lateral offset speed , and the offset distance (D offset ). In this way, this multi-dimensional evaluation mechanism can more comprehensively reflect the actual parking state of the aircraft and avoid misjudgments that may be caused by a single parameter (for example, relying only on the offset distance). For example, under low light conditions, if there is noise interference in image recognition, resulting in a large calculation error of the offset distance D offset , but at this time, the lateral offset speed V ⊥ and the yaw angle Ψ of the aircraft can still maintain accuracy through the dynamic model, and the comprehensive evaluation function F can still effectively judge the state of the aircraft. Another example is that in rainy or snowy weather, if the lateral offset speed V ⊥ suddenly increases due to slippery ground, even if D offset does not exceed the threshold, the system can still trigger an alarm signal in a timely manner through .

[0155] Second, the present invention sets different threshold intervals (F > 0.8, 0.5 < F ≤ 0.8, and F ≤ 0.5). The system can dynamically output corresponding alarm signals according to the risk level: Red alarm (F > 0.8): indicating that the aircraft has a serious attitude deviation or a risk of rapid offset, and it is necessary to immediately stop taxiing or perform manual intervention. Yellow speed limit (0.5 < F ≤ 0.8): prompting the pilot to reduce the taxiing speed to reduce the risk brought by dynamic errors. Green passage (F ≤ 0.5): indicating that the current state is safe and taxiing can continue. In this way, this hierarchical alarm mechanism can effectively prevent the algorithm from being frequently mis-triggered due to fluctuations in a single parameter, while ensuring safety in extreme cases.

[0156] Third, the present invention can adjust the sensitivity of the evaluation function by adjusting the three weight coefficients α, β, and δ according to different application scenarios (for example, different aircraft models, airport layouts, or weather conditions). For example, in sandy dust weather, if the image recognition noise is large and the error of D offset increases, the weight of δ can be reduced, and at the same time, the weights of α and β can be increased (depending on the dynamic model parameters Ψ and V ⊥ ). Another example is that in a low-risk scenario (such as an empty apron), the alarm threshold can be appropriately reduced to improve the system response speed.

[0157] In this embodiment, it should be noted that, first, for the calculation formula of the comprehensive evaluation function, among them, the yaw angle term |Ψ| is used to reflect the angular deviation between the direction of the aircraft and the center line of the apron berth. A larger Ψ value indicates that there is a significant deviation between the actual heading of the aircraft and the target direction (for example, the pilot fails to adjust the direction in time). For the lateral offset speed term , the square term is used to strengthen the negative impact of the lateral speed. For example, even if V⊥ The absolute value is small, but if it persists (e.g., crosswinds cause a sustained shift), the squared term will accumulate and amplify the risk value. Offset distance term D offset Directly quantify the actual spatial deviation between the nose wheel light and the center line.

[0158] In one embodiment of the present invention, step S4 may include:

[0159] Step S4-1: Calculate the direction correction angle:

[0160]

[0161] In the formula, k1 and k2 are PID coefficients, and L remaining Y is the remaining taxiing distance, ΔD is the offset of the aircraft centerline, and ΔD = Y nose -Y centerline Y centerline This represents the coordinates of the centerline of the apron berth on the Y-axis.

[0162] Step S4-2: Calculate the recommended gliding speed:

[0163]

[0164] Step S4-3: Output the calculated direction correction angle and recommended gliding speed.

[0165] In this embodiment, firstly, the direction correction angle θ is used. adj Formula design (combined with remaining gliding distance L) remaining And offset ΔD), so that the magnitude can be dynamically corrected according to the real-time position of the aircraft. Among them, when the offset is large (ΔD>1.5m), the remaining distance L remaining The angle is relatively small, so the direction correction angle θ is small. adj In The angle increases significantly, reminding the pilot to adjust the direction by a larger margin to avoid a final deviation due to insufficient remaining distance. When the deviation is small (ΔD≤0.5m), the direction correction angle θ is... adj The primary reliance is on yaw angle Ψ compensation to ensure the aircraft maintains heading stability as it approaches the target position. For example, under low-light conditions, if image recognition errors cause the calculated value of ΔD to be too large, it can still be adjusted based on the remaining distance L. remaining The weights (k1 terms) are automatically reduced to decrease the correction magnitude to avoid over-adjustment.

[0166] Secondly, this invention recommends a graded taxiing speed strategy, which effectively achieves the following: During high-risk deviations (ΔD>1.5m), low-speed taxiing is recommended to give pilots more time to adjust and reduce the risk of collision. During low-risk deviations (ΔD≤0.5m), higher speed taxiing is recommended to shorten overall parking time and improve airport operational efficiency.

[0167] Third, the formula for calculating the direction correction angle integrates the offset (ΔD) and historical attitude deviation (Ψ) to form a closed-loop control logic. The first term... To achieve rapid response to the current offset, priority is given to correcting major deviations. The second term (k2·Ψ) predicts future offset trends based on the yaw angle change rate (Ψ as a function of time), thus mitigating potential risks in advance.

[0168] Please see Figures 2 to 5 In an exemplary embodiment of the present invention, while outputting the aircraft's speed and forward direction adjustment data according to the aircraft parking guidance model, a corresponding indicator device can also be set in front of the apron parking space. For example, the display content of the indicator device can include five parts, including: the aircraft model currently parking, the adjustment direction, the remaining taxi distance, the T-shaped diagram of the parking space, and the aircraft position diagram.

[0169] Among them, Figure 2 The image shows an A320 aircraft facing the centerline of its parking space, with 29 meters of remaining taxiway.

[0170] exist Figure 3 The image shows an A320 aircraft positioned to the left of the centerline of its parking space, with its direction adjusted to the left and a remaining taxiway of 20 meters.

[0171] exist Figure 4 The image shows an A320 aircraft positioned to the left of the centerline of its parking space, with its direction adjusted to the right. The aircraft has 20 meters of remaining taxiway.

[0172] exist Figure 5 The image shows an A320 aircraft parked in its designated spot.

[0173] In this way, the pilot can more intuitively know the suggested direction for controlling the aircraft, the current attitude of the aircraft, and the remaining taxi distance, so that the pilot can make taxiing adjustments more quickly.

[0174] In one embodiment of the present invention, in step S2-1-2, the color threshold range of the nose wheel light is H∈[0°,30°]∪[330°,360°]; the color threshold range of the left wing light is H∈[0°,15°]; the color threshold range of the right wing light is H∈[90°,150°]; and the saturation threshold is S>0.7.

[0175] In this embodiment, firstly, by setting hue (H) and saturation (S) thresholds, high-precision light segmentation is achieved under adverse weather conditions (such as rain, snow, and low light) based on the optical characteristics of the nose wheel light, left wing light, and right wing light. Specifically, it can resist color distortion interference (for example, in low light environments, the red of the nose wheel light may appear dark red due to ambient light attenuation, but by filtering low saturation noise through a high saturation threshold, it ensures that only the high-saturation target light is retained), and it can also distinguish between the left and right wing lights (by setting corresponding hue thresholds for the left and right wing lights respectively, the left and right wing lights can be accurately distinguished, avoiding misjudgment of left and right positions due to color confusion).

[0176] Secondly, the nose wheel light may have different colors depending on the model (e.g., red, white, or amber). This invention has a wide color threshold range for the nose wheel light, which can be compatible with the recognition of multiple colors.

[0177] In one embodiment of the present invention, step S2-1-1 may further include:

[0178] Step S2-1-1-1: Remove isolated noise points using opening operation:

[0179]

[0180] Step S2-1-1-2: Connect the fractured regions using closing operations:

[0181]

[0182] In the formula, For corrosion operation, For the expansion operation, the structural element W is a 3×3 circular core.

[0183] In this embodiment, firstly, the present invention uses an opening operation (erosion followed by dilation) to remove small, isolated noise points (e.g., raindrops, snow particles, sand reflections, etc.) from the image, preventing them from being misidentified as nose wheel lights or wing lights in step S2-1-2 (color segmentation). In other words, step S2-1-1-1 can achieve the effect of eliminating isolated noise interference and improving the image signal-to-noise ratio. For example, in a rainy scene, the highlight points (isolated white areas) formed by raindrops reflecting the camera light source are filtered by the opening operation, retaining only continuous light areas.

[0184] Secondly, this invention uses a closing operation (expansion followed by erosion) to connect the broken light areas caused by severe weather (such as dust obstruction, rain, or snow). In other words, step S2-1-1-2 can repair the broken target area, ensuring the integrity of the feature. For example, when sand partially obscures the right wing light, the closing operation can connect the segmented green light area, forming a complete outline and ensuring the accuracy of the right wing light coordinate extraction.

[0185] Third, the 3×3 circular core design of the structural element W in this invention balances noise removal and feature preservation. Specifically, the circular core avoids geometric distortion caused by directional structural elements (such as rectangular cores) and adapts to the natural diffusion pattern of light. At the same time, the 3×3 scale effectively removes small-scale noise (e.g., raindrops with a diameter of about 3-5 pixels) without damaging the main structure of the light (e.g., the wing light area is typically larger than 10×10 pixels).

[0186] In one embodiment of the present invention, step S2-1-2 may further include: establishing spatial position constraints for the nose wheel light, left wing light, and right wing light based on the aircraft's geometric features: for the nose wheel light, its spatial position is located in the lower 1 / 3 area of ​​the aircraft parking image; for the left wing light and right wing light, their spatial positions are higher than the nose wheel light and located in the left and right 1 / 4 areas of the aircraft parking image.

[0187] In this embodiment, firstly, the present invention effectively filters out interference sources that are similar in color to the target light but misplaced by spatial position constraints (the nose wheel light is located in the lower 1 / 3 of the image, and the left and right wing lights are located in the left and right 1 / 4 of the image and are higher than the nose wheel light). For example, white ground markings or lights (e.g., runway lights on the ground) may have a hue that overlaps with the nose wheel light, but their position is at the bottom of the image and is excluded by the spatial constraints. As another example, building or vehicle lights (such as green billboards) may be mistaken for wing lights, but are filtered out because their position is off-center from the left and right 1 / 4 of the image.

[0188] Secondly, this invention combines color segmentation and spatial positioning verification to accurately locate light targets. Specifically, for nose wheel light positioning, the nose wheel light is actually located below the front of the fuselage, and its mapping in the image conforms to the lower 1 / 3 region constraint, avoiding misidentification of white lights in the middle of the fuselage (e.g., landing lights) as nose wheel lights. For wing light positioning, the left / right wing lights are actually located at the wingtips, and during taxiing, they are distributed in the left and right 1 / 4 regions of the image, higher than the nose wheel light, which conforms to geometric logic.

[0189] Third, even in adverse weather conditions, where color segmentation fails (e.g., rain or snow reduces saturation), spatial constraints still provide redundancy for verification. For example, if the nose wheel light's color segmentation is incomplete due to low light, its position is still located in the lower 1 / 3 region, and spatial constraints can assist in positioning. Similarly, if the left wing light is partially obscured, its remaining area is still located in the left 1 / 4 region, and feature points in that area can be repaired first.

[0190] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for guiding apron berths based on image recognition technology, characterized in that, include: Step S1: In response to the parking signal issued by the aircraft, acquire the aircraft parking image, the aircraft's current speed and direction of travel, wherein the aircraft parking image includes the aircraft to be parked and the center line of the apron berth; Step S2: Identify the nose wheel light, left wing light, and right wing light of the aircraft to be parked in the aircraft parking image, and establish the aircraft's attitude model in conjunction with the center line of the apron parking space; establish the aircraft's speed model in conjunction with the aircraft's current speed and direction of travel. Step S3: Establish an aircraft parking guidance model based on the aircraft's attitude and velocity models; Step S4: Output the aircraft's speed and direction of travel adjustment data based on the aircraft's parking guidance model; Step S2 includes: Establish the aircraft's attitude model: Step S2-1-1: Convert the image obtained in step S1-2 to HSV space and separate the hue (H) and saturation (S) channels: ; In the formula, This is the normalized value for the red channel. This is the normalized value for the green channel. This is the normalized value for the blue channel; Step S2-1-2: Perform image segmentation based on the color threshold range and saturation threshold range of the nose wheel light, left wing light and right wing light to identify the nose wheel light, left wing light and right wing light of the aircraft to be parked in the aircraft parking image respectively. Step S2-1-3: Combining the centerline of the apron berth in the aircraft parking image and the known wingspan parameters of the aircraft, construct the spatial mapping relationship between the aircraft body coordinate system and the coordinate system of the apron berth centerline; Step S2-1-4: Use the perspective projection transformation model to transform the two-dimensional image coordinates into three-dimensional spatial coordinates: In the formula, For the camera intrinsic parameter matrix, Image pixel coordinates, For depth estimates and , This refers to the wingspan of the aircraft. Let X be the coordinate of the right wing light on the X-axis. The coordinates of the left wing light on the X-axis; Step S2-1-5: Calculate the aircraft's pitch angle using the geometric constraints between the spatial triangle formed by the coordinates of the nose wheel light, left wing light, and right wing light and the centerline of the apron parking space. Roll angle and yaw angle : , , In the formula, The three-dimensional coordinates of the left wing light are: The three-dimensional coordinates of the right wing light are: The coordinates of the nose wheel light. Let X be the coordinates of the apron berth on the X-axis. This refers to the fuselage length; Step S2 further includes: Establish an aircraft speed model: Step S2-2-1: Perform velocity vector decomposition according to the following formula: In the formula, For the nose wheel light in time Position coordinates at time For the nose wheel light in time Position coordinates at time The time interval between consecutive frames. Yaw angular velocity; Step S2-2-2: Calculate the effective taxiing speed and lateral offset speed of the aircraft along the centerline of the apron berth according to the following formula: In the formula, The effective taxiing speed of the aircraft along the centerline of the apron parking space. This represents the aircraft's lateral drift velocity; Step S3 includes: Establish an aircraft parking guidance model: Step S3-1: Define the comprehensive evaluation function: In the formula, , and These are the weighting coefficients, The offset distance between the nose wheel light and the center line of the apron berth and , The coordinates of the nose wheel light on the Y-axis; Step S3-2: Output the corresponding instruction based on the value of the comprehensive evaluation function: when When this occurs, a red alarm signal is output; when At that time, a yellow speed limit warning signal will be output; when At that time, a green passage signal will be output.

2. The apron berth guidance method based on image recognition technology according to claim 1, characterized in that, Step S1 further includes: Step S1-1: Perform spectral normalization on the acquired aircraft parking image according to the following formula: In the formula, For the original image in coordinates Color channel pixel values, For channel The global mean, For channel standard deviation For channel Gain coefficient, For channel The bias term; Step S1-2: Spatial sharpening of the image obtained in step S1-1 is performed according to the following formula: In the formula, For the Laplace operator, The standard deviation is Gaussian kernel, This is the sharpening weight coefficient.

3. The apron berth guidance method based on image recognition technology according to claim 1, characterized in that, Step S4 includes: Step S4-1: Calculate the direction correction angle: In the formula, , For PID coefficients, The remaining gliding distance. This represents the offset of the aircraft's centerline. , This represents the coordinates of the centerline of the apron berth on the Y-axis. Step S4-2: Calculate the recommended gliding speed: ； Step S4-3: Output the calculated direction correction angle and recommended gliding speed.

4. The apron berth guidance method based on image recognition technology according to claim 1, characterized in that, In step S2-1-2, The color threshold range of the nose wheel light is: ; The color threshold range for the left wing light is: ; The color threshold range for the right wing light is: ; The saturation threshold is: .

5. The apron berth guidance method based on image recognition technology according to claim 1, characterized in that, Step S2-1-1 further includes: Step S2-1-1-1: Remove isolated noise points using opening operation: ； Step S2-1-1-2: Connect the fractured regions using closing operations: In the formula, For corrosion operation, For expansion operations, structural elements It has a 3×3 circular core.

6. The apron berth guidance method based on image recognition technology according to claim 1, characterized in that, Step S2-1-2 further includes: Based on the aircraft's geometric characteristics, establish spatial positional constraints for the nose wheel light, left wing light, and right wing light: For the nose wheel light, its spatial position is located in the lower 1 / 3 area of ​​the aircraft parking image; The left and right wing lights are positioned higher than the nose wheel lights and within the left and right quarters of the aircraft's parking image.

Images