Airport floor lamp screw looseness detection method based on machine vision

Abstract

The invention discloses an airport floor lamp screw looseness detection method based on machine vision. The airport floor lamp screw looseness detection method comprises the following steps: S1, marking a fastened screw by using thread marking glue; s2, performing image acquisition on a floor lamp area through a vehicle-mounted laser line scanning camera, and processing an original image; s3, floor lamp positioning is carried out; s4, screw positioning is carried out; s5, constructing a classifier through the residual neural network, and judging the state of the screw; and S6, outputting a result, and reporting the position information of the fault screw. An automatic image acquisition and processing system is established, human input can be greatly reduced, a traditional lamp-by-lamp inspection mode is upgraded to area coverage type scanning, the requirement of an airport for safe and efficient facility detection is met, meanwhile, a screw looseness judgment model is established based on a deep learning feature recognition network, and the safety and reliability of the airport are improved. Judgment is made according to the obtained screw depth point cloud information and color information, interference of human subjective factors is avoided, and the missing detection rate and the false detection rate are effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of aviation transportation technology, and in particular to a method for detecting loose screws of airport floor lamps based on machine vision. Background Art

[0002] In the aviation industry, airport ground lights serve as critical facilities for guiding aircraft takeoff, landing, and taxiing, and their proper operation is directly related to flight safety. Typically installed on airport runways and taxiways, these lights are subject to constant vibration from aircraft takeoff and landing, complex weather conditions, and vehicle overrun. These forces can easily cause the screws holding them to loosen. Once loosened, these screws can cause the lights to shift, tilt, or even fall off. This not only affects the light's illumination angle and guidance effectiveness, but can also cause safety incidents such as scratches during takeoff and landing, posing a serious threat to aviation safety.

[0003] Currently, the detection of loose screws in airport floor lights mainly relies on manual inspections. Inspectors need to regularly check each floor light within the airport area, using visual observation or simple tools (such as wrenches) to determine whether the screws are loose. This manual inspection method has obvious limitations. First, the number of airport floor lights is huge and their distribution is wide. Inspectors need to check the screws of each floor light one by one. This one-by-one inspection method not only requires a large investment of manpower costs, but also consumes a lot of time, resulting in extremely low overall inspection efficiency and failing to meet the airport's needs for safe and rapid facility inspections. Second, manual inspections can usually only be carried out at night after flights are stopped. Traditional visual inspection equipment is prone to problems such as excessive image noise and blurred edge features due to insufficient lighting in dim conditions at night. Even with auxiliary lighting equipment, it is difficult to avoid strong light reflections or shadow interference, making it difficult to identify subtle loose screw features. At the same time, manual visual fatigue increases in nighttime working environments, further reducing detection accuracy and making missed detections or false detections more likely. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for detecting loose screws of airport floor lamps based on machine vision.

[0005] The purpose of the present invention is achieved through the following technical solutions: A method for detecting loose screws of airport floor lights based on machine vision, comprising the following steps:

[0006] S1: Use thread marking glue to mark the screws after tightening;

[0007] S2: Use the vehicle-mounted laser line scan camera to capture images of the ground lamp area and process the original images;

[0008] S3: Use the YOLOv12 target detection framework to process the fusion data of grayscale image and depth map to locate the ground lamp;

[0009] S4: Based on the ground lamp ROI area output in step S3, screw positioning is performed according to the YOLOv12 multi-scale detection head architecture;

[0010] S5: Build a classifier through residual neural network to judge the status of screws;

[0011] S6: Output the result and report the location information of the faulty screw.

[0012] Preferably, in step S2, processing the original image specifically includes the following steps:

[0013] S21: Use an inertial measurement unit to record device motion parameters in real time, and perform spatial coordinate transformation on the original point cloud data through a kinematic model to eliminate the initial source of streak noise;

[0014] S22: Perform global surface fitting on the depth image based on the thin plate spline function and optimize the control point weights using the least squares method;

[0015] S23: Perform wavelet packet decomposition on the difference matrix between the original depth map and the fitted surface to retain the effective signal of the key features of the nut edge;

[0016] S24: Normalization processing is used to uniformly adjust and standardize the image depth change features.

[0017] Preferably, step S3 further includes the following steps:

[0018] S31: Fusion of the depth map with the temporal and spatial alignment algorithm into a dual-channel input tensor, and use of depth information to enhance the distinction between ground lights and road surfaces;

[0019] S32: Train the model and inject salt and pepper noise to simulate interference during acquisition.

[0020] S33: Reduce the number of parameters through grouped convolution and cross-stage connections, and embed dynamic routing attention in the Backbone stage to adjust the feature fusion weights of grayscale and depth maps according to the input image;

[0021] S34: Filter road surface depressions through the adaptive NMS algorithm and output the positioning coordinates and length and width information of the ground light target.

[0022] Preferably, in step S32, the backbone network of the model adopts a CSPNeXt structure combined with an ELAN-S module.

[0023] Preferably, step S4 further includes the following steps:

[0024] S41: Based on the floor lamp position information obtained in step S34, crop the floor lamp image from the original size image and scale it to a size of 512×512;

[0025] S42: Feature enhancement of the cropped pixel image through deformable convolution;

[0026] S43: The model introduces a feature pyramid network for cross-scale feature fusion and outputs the positioning coordinates and length and width information of the screw target.

[0027] Preferably, in step S5, the screw status includes three states: tightened, loose, and mark missing.

[0028] The present invention has the following advantages: the present invention builds an automated image acquisition and processing system, which can greatly reduce manpower input, upgrade the traditional light-by-light inspection mode to area coverage scanning, and meet the airport's needs for safe and efficient facility inspection. At the same time, a screw loosening judgment model is established based on a deep learning feature recognition network, and judgment is made according to the acquired screw depth point cloud information and color information, avoiding interference from human subjective factors and effectively reducing missed detection and false detection rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a structural diagram of the process of detecting loose screws of airport ground lights based on machine vision. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0031] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are also within the scope of protection of the present invention.

[0032] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other.

[0033] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0034] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the inventive product is typically placed when in use, or are the orientations or positional relationships commonly understood by those skilled in the art. These terms are intended only to facilitate the description of the present invention and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0035] In the description of the present invention, it should also be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0036] In this embodiment, if Figure 1 As shown, a method for detecting loose screws of airport floor lights based on machine vision includes the following steps:

[0037] S1: Use thread marking glue to mark the tightened screws. Specifically, thread marking glue is a marking material used for threaded connections. It forms a clear mark on the bolt or nut so that it can be identified through visual inspection when the thread is loose or disassembled. This colloid has good adhesion and can remain stable in various environments, such as water, oil, gasoline and some organic solvents. It is also resistant to diluted alkali and acid, ensuring the durability of the mark. Therefore, the maintenance interval of this step can be very long, and marking is not required before every inspection.

[0038] S2: Use the vehicle-mounted laser line scan camera to capture images of the ground light area and process the original images. Specifically, the acquisition width is more than 10 meters, and the speed can reach more than 60km / h, which can complete runway and taxiway data collection in a relatively short time.

[0039] S3: Use the YOLOv12 target detection framework to process the fusion data of grayscale image and depth map to locate the ground lamp;

[0040] S4: Based on the ground lamp ROI area output in step S3, screw positioning is performed according to the YOLOv12 multi-scale detection head architecture;

[0041] S5: A classifier is constructed through a residual neural network to determine the state of the screw. Preferably, the screw state includes three states: tightened, loose, and missing mark. Specifically, the classifier is constructed based on the existing method, which is not improved here and will not be described in detail.

[0042] S6: Output the results and report the location of the faulty screw. This automated image acquisition and processing system significantly reduces manpower input, upgrading the traditional light-by-light inspection model to area-wide scanning, meeting airports' needs for safe and efficient facility inspections. A screw loosening determination model is established based on a deep learning-based feature recognition network. This model uses the acquired screw depth point cloud and color information to make judgments, eliminating interference from subjective factors and effectively reducing missed and false detection rates.

[0043] Furthermore, in step S2, processing the original image specifically includes the following steps:

[0044] S21: Use an inertial measurement unit to record device motion parameters in real time, and perform spatial coordinate transformation on the original point cloud data through a kinematic model to eliminate the initial source of streak noise;

[0045] S22: Perform global surface fitting on the depth image based on the thin plate spline function and optimize the control point weights using the least squares method;

[0046] S23: Perform wavelet packet decomposition on the difference matrix between the original depth map and the fitted surface to retain the effective signal of the key features of the nut edge;

[0047] S24: Normalization processing is used to uniformly adjust and standardize the image depth change features. Specifically, due to external factors such as light and environment or defects in the acquisition equipment itself, the collected images often have uneven depth and abnormal points. In order to make the collected images clean, clear and easy for neural network recognition, the original images need to be further processed. In step S21, motion distortion pre-correction is performed for the speed fluctuation (±5% error) and platform vibration (amplitude ≤ 0.5mm) during the movement of the laser line scan camera. That is, the inertial measurement unit (IMU) is used to record the equipment motion parameters in real time, and the original point cloud data is transformed into spatial coordinates through the kinematic model to eliminate the initial source of stripe noise caused by unstable motion. In step S22, the depth image is fitted with a global surface using a thin plate spline function to construct a dynamic grid model containing 128×128 control points. The control point weights are optimized by the least squares method so that the fitted surface can adapt to the surface curvature of the ground lamp and filter out periodic motion noise. In step S23, the difference matrix between the original depth map and the fitted surface is decomposed by wavelet packet, and adaptive thresholds based on noise energy distribution are set in 8 frequency bands (using 3 The image processing principle is dynamic calculation, which focuses on suppressing high-frequency noise (grayscale fluctuation ≥ 150) caused by strong light reflections from the airport and medium-frequency artifacts (frequency 5-20 Hz) induced by vibration, while retaining the effective signals of key features such as the nut edge. Step S24 performs normalization processing to ensure consistency and comparability of image data under different conditions.

[0048] Furthermore, step S3 further includes the following steps:

[0049] S31: Fusion of the depth map with the temporal and spatial alignment algorithm into a dual-channel input tensor, and use of depth information to enhance the distinction between ground lights and road surfaces;

[0050] S32: Train the model and simulate the interference during acquisition by injecting salt and pepper noise; preferably, the backbone network of the model adopts the CSPNeXt structure combined with the ELAN-S module.

[0051] S33: Reduce the number of parameters through grouped convolution and cross-stage connections, and embed dynamic routing attention in the Backbone stage to adjust the feature fusion weights of grayscale and depth maps according to the input image;

[0052] S34: Filter road surface depressions using the adaptive NMS algorithm and output the location coordinates, length, and width of the ground light target. Specifically, step S32 improves the model's versatility and anti-interference capabilities. In step S33, the weight range is 0.3–0.7; in step S34, the threshold is set to 0.9.

[0053] In this embodiment, step S4 further includes the following steps:

[0054] S41: Based on the floor lamp position information obtained in step S34, crop the floor lamp image from the original size image and scale it to a size of 512×512;

[0055] S42: Feature enhancement of the cropped pixel image through deformable convolution;

[0056] S43: The model incorporates a feature pyramid network for cross-scale feature fusion, outputting the location coordinates and length and width information of the screw target. Specifically, by locating the ground lights and screws, it can avoid false detection of screws in locations other than ground lights on the airport pavement.

[0057] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting loose screws in airport floor lights based on machine vision, characterized by: The following steps are involved: S1: Use thread marking glue to mark the screws after tightening; S2: Use the vehicle-mounted laser line scan camera to capture images of the ground lamp area and process the original images; S3: Use the YOLOv12 target detection framework to process the fusion data of grayscale image and depth map to locate the ground lamp; S4: Based on the ground lamp ROI area output in step S3, screw positioning is performed according to the YOLOv12 multi-scale detection head architecture; S5: Build a classifier through residual neural network to judge the status of screws; S6: Output the result and report the location information of the faulty screw.

2. The method for detecting loose screws of airport floor lights based on machine vision according to claim 1, characterized in that: In step S2, processing the original image specifically includes the following steps: S21: Use an inertial measurement unit to record device motion parameters in real time, and perform spatial coordinate transformation on the original point cloud data through a kinematic model to eliminate the initial source of streak noise; S22: Perform global surface fitting on the depth image based on the thin plate spline function and optimize the control point weights using the least squares method; S23: Perform wavelet packet decomposition on the difference matrix between the original depth map and the fitted surface to retain the effective signal of the key features of the nut edge; S24: Normalization processing is used to uniformly adjust and standardize the image depth change features.

3. The method for detecting loose screws of airport floor lights based on machine vision according to claim 2, characterized in that: The step S3 further includes the following steps: S31: Fusion of the depth map with the temporal and spatial alignment algorithm into a dual-channel input tensor, and use of depth information to enhance the distinction between ground lights and road surfaces; S32: Train the model and inject salt and pepper noise to simulate interference during acquisition. S33: Reduce the number of parameters through grouped convolution and cross-stage connections, and embed dynamic routing attention in the Backbone stage to adjust the feature fusion weights of grayscale and depth maps according to the input image; S34: Filter road surface depressions through the adaptive NMS algorithm and output the positioning coordinates and length and width information of the ground light target.

4. The machine vision-based airport floor light screw loosening detection method according to claim 3, characterized in that: In step S32, the backbone network of the model adopts the CSPNeXt structure combined with the ELAN-S module.

5. The method for detecting loose screws of airport floor lights based on machine vision according to claim 4, characterized in that: The step S4 further includes the following steps: S41: Based on the floor lamp position information obtained in step S34, crop the floor lamp image from the original size image and scale it to a size of 512×512; S42: Feature enhancement of the cropped pixel image through deformable convolution; S43: The model introduces a feature pyramid network for cross-scale feature fusion and outputs the positioning coordinates and length and width information of the screw target.

6. The method for detecting loose screws of airport floor lights based on machine vision according to claim 5, characterized in that: In step S5, the screw status includes three states: tightened, loose, and mark missing.

Images