Method for detecting frost on the surface of an aircraft based on laser thickness measurement and visual image analysis

Abstract

The application provides a kind of aircraft surface frost detection method based on laser thickness measurement and visual image analysis, which comprises the following steps: S1, build aircraft surface frost detection system;S2, train and build image feature reference library under ice-free state;S3, initialize setting to detection system;S4, calculate frost layer thickness;S5, collect aircraft surface image and calculate abnormal value;S6, collect image of ground area and calculate abnormal value;S7, normalize and weighted fusion calculation are carried out to multi-source data;S8, judge whether the fusion value is greater than the preset threshold, if the fusion value is greater than the preset threshold, there is ice and frost in the current detection area of aircraft surface, and the detection system will trigger defrosting prompt alarm and defrosting. The application constructs a multi-modal fusion scheme based on laser scattering model and visual image anomaly detection algorithm, considers the complex mechanism and optical properties of frost layer formation, this method has higher robustness to environmental interference, and can trigger alarm and deicing operation in time, effectively reduces the burden of aircraft maintenance personnel, enhances the safety and efficiency of aircraft ground operation.

Description

Technical Field

[0001] This invention relates to the field of aviation inspection technology, and in particular to a method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis. Background Technology

[0002] Frost buildup on aircraft surfaces is a long-standing and significant safety concern in the civil aviation industry. Especially in low-temperature, high-humidity, or snowy environments, frost deposits on the fuselage can affect the aircraft's aerodynamic characteristics, increasing drag, reducing lift, and in severe cases, potentially leading to control failure or takeoff disruption. Therefore, civil aviation authorities in various countries have imposed strict requirements on ensuring that aircraft surfaces are free of frost before takeoff; critical areas must be free of frost before takeoff.

[0003] Currently, the detection of frost on aircraft surfaces mainly relies on manual visual inspection and inference from traditional meteorological data. Manual inspection relies on experience-based judgment, which is inefficient, susceptible to subjective errors, and cannot achieve real-time monitoring. While meteorological data-based prediction methods can provide some indication of frost formation trends, they cannot directly reflect the actual thickness and distribution of frost on the aircraft surface.

[0004] With the increasing demand for aircraft surface frost detection, much research has focused on single-modal sensor detection, such as deep learning algorithms based on visible light images and temperature anomaly analysis based on infrared radiation. However, these methods generally suffer from insufficient generalization ability, low environmental adaptability, and poor real-time performance, making it difficult to meet the needs of efficient airport scheduling and intelligent operation and maintenance. Therefore, how to integrate multiple sensing technologies, overcome the limitations of single sensors, and achieve high-precision and robust frost detection technology has become an important research direction for aviation safety assurance. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis. Addressing the needs of frost detection on civil aircraft surfaces, a multimodal fusion scheme is constructed based on a laser scattering model and a visual image anomaly detection algorithm. This scheme fully considers the complex mechanism and optical characteristics of frost formation and has the potential for widespread application in rapid ground inspection before and after civil aircraft parking or flight, providing support for the construction of smart airports.

[0006] On one hand, the present invention provides a method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis, which includes the following steps: S1. Set up an aircraft surface frost detection system and configure the system for training operation mode. S2. Conduct offline training on the detection system, collect aircraft surface image samples under frost-free and normal environmental conditions, and construct an image feature benchmark library for aircraft surface under frost-free conditions. S3. Set the detection system to detection mode and initialize the detection system. S4. Calculate the frost thickness in the target detection area using the laser reflection principle and light intensity attenuation model; S5. Acquire images of the aircraft surface in the current detection area using a color camera, extract image features, and compare them with the image feature benchmark library of the aircraft surface under frost-free conditions constructed in step S2 to calculate the outlier value in the current detection area. S6. Acquire images of the ground area using a color camera and calculate outliers in the ground area; S7. Perform data normalization on the frost layer thickness calculated in step S4, the aircraft surface anomaly value calculated in step S5, and the ground area anomaly value calculated in step S6. Then, perform weighted fusion on the normalization results and calculate the final fused value. The calculation formula is as follows: ; In the formula, This indicates the fusion weight of the laser thickness measurement channel. The fusion weights represent outliers in aircraft surface images. The fusion weights represent outliers in the ground region. This represents the normalized laser-measured frost thickness value. This represents outliers in the normalized aircraft surface image. Indicates outliers in the normalized ground image; S8. Determine whether the fusion value is greater than the preset threshold. If the fusion value is less than or equal to the preset threshold, there is no frost in the current detection area on the aircraft surface. Then return to step S4 to perform the next round of detection on the aircraft surface. If the fusion value is greater than the preset threshold, there is frost in the current detection area on the aircraft surface. The detection system will trigger a defrosting warning and perform defrosting.

[0007] Preferably, the laser reflection acquisition in step S4, the aircraft surface image acquisition in step S5, and the ground area image acquisition in step S6 are performed simultaneously and recorded with the same timestamp in the detection system.

[0008] Furthermore, step S1 specifically involves the following steps: S11. Arrange the laser, black and white camera, color camera and temperature and humidity sensor on the detection system bracket respectively, and place the detection system bracket on one side of the aircraft wing. S12. Adjust the incident angle of the laser on the aircraft surface; S13. Adjust the field of view of the black-and-white camera and the color camera on the aircraft surface and the ground area. S14. Connect the laser, black and white camera, color camera and temperature and humidity sensor to the remote server of the detection system respectively. S15. Configure a time synchronization mechanism in the detection system so that laser measurement, image acquisition and environmental detection parameter acquisition can be completed synchronously in each acquisition cycle; S16. Test the data integrity of each data transmission channel and set the operating mode of the detection system to training mode.

[0009] Furthermore, step S2 specifically involves the following steps: S21. Acquire images of the aircraft surface under normal environmental conditions without ice or frost. S22. Repeat step S21 under different time periods and different lighting conditions to cover the range of typical visual changes on the aircraft surface and ground area. S23. Mark all image data on the aircraft surface and ground area collected in step S22 as being in an anomaly-free state and use them as training samples. S24. Use a preset image feature extraction algorithm to encode each acquired image; S25. Divide each image into multiple image blocks and use a pre-trained convolutional neural network model to calculate the feature vectors corresponding to different image blocks; S26. Store the feature vectors of all image blocks into the image feature reference library in regional order, and maintain the correspondence with the original image.

[0010] Furthermore, step S3 specifically involves the following steps: S31. Set the optical parameters of the laser; S32, Set the resolution, exposure time, and brightness of the auxiliary light source for the black-and-white camera and the color camera; S33. Select a fixed detection area on the aircraft surface and set the scattering coefficient, porosity correction coefficient, and frost detection threshold.

[0011] Furthermore, step S4 specifically involves the following steps: S41. Start the laser, and the laser will irradiate the target detection area at the set incident angle; S42. After the black and white camera receives the reflected signal from the surface of the aircraft, it processes the signal through a filter to obtain an effective image signal in a single band. S43. Extract the light intensity value corresponding to the effective image in step S42, that is, the reflected light intensity actually collected by the black and white camera. S44. Calculate the intensity of the transmitted light after the laser passes through the frost layer. The expression is: ; In the formula, This represents the intensity of transmitted light under ideal conditions. Indicates the intensity of the incident light. Represents an exponential function. Indicates the scattering coefficient of the frost layer. Indicates the thickness of the frost layer; S45. Define the diffuse reflectance of laser light by a frost layer, and its expression is: ; S46. Construct a linear approximation model under thin-layer conditions, that is, in Under the given conditions, the linear approximation model is: ; In the formula, express higher-order infinitesimals; S47. Substitute the actual reflected light intensity collected in step S43 into the linear approximation model to calculate the frost layer thickness, the expression of which is: ; In the formula, This represents the intensity of the reflected light obtained from actual measurement; S48. Obtain multiple laser measurement values ​​and calculate the average thickness of the frost layer.

[0012] Preferably, step S5 specifically includes the following steps: S51. Acquire color images of the aircraft surface in the current detection area using a color camera; S52. Divide the color image of the aircraft surface acquired in step S51 into multiple image blocks, and use a pre-trained convolutional neural network model to calculate the feature vectors corresponding to different image blocks. S53. Call the image feature benchmark library of the aircraft surface under frost-free conditions built during offline training; S54. Calculate the outlier values ​​on the aircraft surface in the current detection area.

[0013] Furthermore, step S54 specifically involves the following steps: S541. Calculate the minimum Euclidean distance between the feature vector of each image patch in the acquired color image of the aircraft surface and the feature vector of the image under frost-free conditions. The expression is as follows: ; In the formula, The feature vector representing an image patch. This represents the image feature vector in a frost-free state. S542. The nearest neighbor feature matching and outlier calculation method is used to perform feature matching calculation on the entire aircraft surface image acquired. S543. Aggregate the outliers of all image patches and calculate the outliers of the entire image. The expression is as follows: ; In the formula, This represents the outlier results for the entire surface of the aircraft. This represents the set of image scales (e.g., different resolution levels) participating in the aggregation. This indicates the current scale level under consideration. Indicated in scale The total number of image blocks divided into subdivisions, Indicates the first Image content of each image block Representing scale Next, the Outliers in an image patch Representing scale The average outlier value of all image patches below. Representing scale Weighting coefficients; S544. Record and store the abnormal values ​​on the surface of the aircraft in the current detection area.

[0014] Furthermore, step S6 specifically involves the following steps: S61. Acquire images of the ground area using a color camera; S62. Input the acquired ground images directly into the convolutional neural network model; S63. The model performs an overall analysis of the input image and outputs outliers in the ground area.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The aircraft surface frost detection method based on laser thickness measurement and visual image analysis provided by this invention addresses the needs of frost detection on civil aircraft surfaces by constructing a multimodal fusion scheme. It fully considers the complex mechanism and optical characteristics of frost formation and establishes a linear relationship between frost thickness and diffuse reflection light intensity based on the modified Beer-Lambert law, thereby achieving real-time frost thickness detection. At the same time, the multi-scale visual feature extraction algorithm based on convolutional neural networks can quickly identify large-scale frost coverage and local icing anomalies.

[0016] 2. The aircraft surface frost detection method based on laser thickness measurement and visual image analysis provided by this invention realizes frost monitoring by linearly weighted fusion of two types of detection information: frost thickness and diffuse reflection light intensity. This system has higher robustness to environmental interference and can trigger alarms and de-icing operations in a timely manner, thereby effectively reducing more human judgment in aircraft maintenance procedures and enhancing the safety and efficiency of aircraft ground operations.

[0017] 3. The aircraft surface frost detection method based on laser thickness measurement and visual image analysis provided by this invention can maintain good stability in the apron environment with large temperature fluctuations and varying lighting conditions. It has the potential to be widely used in rapid ground inspection before and after civil aircraft parking or flight, and can provide support for the construction of smart airports. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the aircraft surface frost detection method based on laser thickness measurement and visual image analysis according to the present invention. Figure 2 This is a flowchart of the aircraft surface frost detection method according to Embodiment 1 of the present invention; Figure 3 Here is a system structure diagram of the aircraft surface frost detection method according to Embodiment 1 of the present invention: Figure 4 Here is a structural diagram of the visual outlier algorithm in Embodiment 2 of the present invention: Figure 5 This is a structural diagram of the fusion decision algorithm in Embodiment 2 of the present invention; Figure 6 This is a graph showing the frost layer thickness data measured in Embodiment 2 of the present invention; Figure 7 This is a graph showing the proportion of frosted area measured in Embodiment 2 of the present invention; Figure 8 This is an RGB image acquired in Embodiment 2 of the present invention; Figure 9 This is a visualization rendering of the frost thickness measured in Embodiment 2 of the present invention; Figure 10 A line graph comparing single radar modal measurement, single visual modal measurement, and fused measurement; Figure 11 This is a schematic diagram of the ablation mean square error histogram. Detailed Implementation

[0019] This invention provides a method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis, such as... Figure 1 , 2 and Figure 3 As shown, it includes the following steps: S1. Set up an aircraft surface frost detection system. The detection system is set to training mode. The specific steps are as follows: S11. Arrange the laser, black and white camera, color camera and temperature and humidity sensor on the detection system bracket respectively, and place the detection system bracket on one side of the aircraft wing. S12. Adjust the incident angle of the laser on the aircraft surface; S13. Adjust the field of view of the black-and-white camera and the color camera on the aircraft surface and the ground area. S14. Connect the laser, black and white camera, color camera and temperature and humidity sensor to the remote server of the detection system respectively. S15. Configure a time synchronization mechanism in the detection system so that laser measurement, image acquisition and environmental detection parameter acquisition can be completed synchronously in each acquisition cycle; S16. Test the data integrity of each data transmission channel and set the operating mode of the detection system to training mode.

[0020] S2. Conduct offline training on the detection system, collect aircraft surface image samples under frost-free and normal environmental conditions, and construct an image feature benchmark library for aircraft surfaces under frost-free conditions, where frost-free means zero frost coverage. The specific steps are as follows: S21. Acquire images of the aircraft surface under normal environmental conditions without ice or frost. S22. Repeat step S21 under different time periods and different lighting conditions to cover the range of typical visual changes on the aircraft surface and ground area. S23. Mark all image data on the aircraft surface and ground area collected in step S22 as being in an anomaly-free state and use them as training samples. S24. Use a preset image feature extraction algorithm to encode each acquired image; S25. Divide each image into multiple image blocks and use a pre-trained convolutional neural network model to calculate the feature vectors corresponding to different image blocks; S26. Store the feature vectors of all image blocks into the image feature reference library in regional order, and maintain the correspondence with the original image.

[0021] S3. The detection system is put into operation. The system's operating mode is switched to detection mode, and initial settings are performed. The specific steps are as follows: S31. Set the optical parameters of the laser; S32, Set the resolution, exposure time, and brightness of the auxiliary light source for the black-and-white camera and the color camera; S33. Select a fixed detection area on the aircraft surface and set the scattering coefficient, porosity correction coefficient, and frost detection threshold.

[0022] S4. Calculate the frost layer thickness in the target detection area using the laser reflection principle and light intensity attenuation model. The specific steps are as follows: S41. Start the laser, and the laser will irradiate the target detection area at the set incident angle; S42. After the black and white camera receives the reflected signal from the surface of the aircraft, it processes the signal through a filter to obtain an effective image signal in a single band. S43. Extract the light intensity value corresponding to the effective image in step S42, that is, the reflected light intensity actually collected by the black and white camera. S44. According to Beer-Lambert's law, calculate the transmitted light intensity after the laser passes through the frost layer. The expression is as follows: ; In the formula, This represents the intensity of transmitted light under ideal conditions. Indicates the intensity of the incident light. Represents an exponential function. Indicates the scattering coefficient of the frost layer. Indicates the thickness of the frost layer; S45. Define the diffuse reflectance of laser light by a frost layer, and its expression is: ; S46. Construct a linear approximation model under thin-layer conditions, that is, in Under the given conditions, the linear approximation model is: ; In the formula, express higher-order infinitesimals; S47. Substitute the actual reflected light intensity collected in step S43 into the linear approximation model to calculate the frost layer thickness, the expression of which is: ; In the formula, This represents the intensity of the reflected light obtained from actual measurement; S48. Obtain multiple laser measurement values ​​and calculate the average thickness of the frost layer.

[0023] S5. Acquire an image of the aircraft surface in the current detection area using a color camera, further extract its image features, and compare it with the image feature benchmark library of the aircraft surface under frost-free conditions constructed in step S2 to calculate the outlier value of the current detection area. The specific steps are as follows: S51. Acquire color images of the aircraft surface in the current detection area using a color camera; S52. Divide the color image of the aircraft surface acquired in step S51 into multiple image blocks, and use a pre-trained convolutional neural network model to calculate the feature vectors corresponding to different image blocks. S53. Call the image feature benchmark library of the aircraft surface under frost-free conditions built during offline training; S54. Calculate the outlier values ​​on the aircraft surface in the current detection area. The specific steps are as follows: S541. Calculate the minimum Euclidean distance between the feature vector of each image patch in the acquired color image of the aircraft surface and the feature vector of the image under frost-free conditions. The expression is as follows: ; In the formula, The feature vector representing an image patch. This represents the image feature vector in a frost-free state. S542. Using the nearest neighbor feature matching and outlier calculation method, feature matching calculation is performed on the entire acquired aircraft surface image. The expression is as follows: ; In the formula, This represents the currently acquired image of the aircraft's surface. This represents the set after dividing the image into multiple image blocks. It is a feature vector extracted from a specific image patch, used to characterize the texture and structural information of that local region. It is a reference feature vector in a normal image feature library established under frost-free conditions, representing the image feature distribution under ideal conditions. Indicates the current test feature The most similar normal feature vector is used to measure whether the current region deviates from the normal state. This is the image patch in the entire test image that differs the most from normal features, and is the location most likely to have frost layer anomalies. S543. Aggregate the outliers of all image patches and calculate the outliers of the entire image. The expression is as follows: ; In the formula, This represents the outlier results for the entire aircraft surface, used to quantify the overall degree of anomalies in the current image. This represents the set of image scales participating in the aggregation, such as different resolution levels. This indicates the current scale level under consideration. Indicated in scale The total number of image blocks divided into subdivisions, Indicates the first Image content of each image block Representing scale Next, the Outliers in an image patch Representing scale The average outlier value of all image patches below. Representing scale The weighting coefficients are used to control the contribution ratio of different scales to the total outliers; S544. Record and store the abnormal values ​​on the surface of the aircraft in the current detection area.

[0024] S6. Acquire images of the ground area using a color camera and calculate the outliers in the ground area. The specific steps are as follows: S61. Acquire images of the ground area using a color camera; S62. The collected ground images are directly input into the convolutional neural network model; S63. The model performs an overall analysis of the input image and outputs outliers in the ground area.

[0025] S7. Perform data normalization on the frost layer thickness calculated in step S4, the aircraft surface anomaly value calculated in step S5, and the ground area anomaly value calculated in step S6. Then, perform weighted fusion on the normalization results and calculate the final fused value. The calculation formula is as follows: ; In the formula, This represents the fusion weight of the laser thickness measurement channel, used to control the degree of influence of the frost layer thickness on the final fusion value. The fusion weights represent outliers in the aircraft surface image, reflecting the importance of image channels in the final judgment. The fusion weights represent outliers in the ground region, used to help correct for the impact of environmental interference on the detection results. This represents the normalized laser-measured frost thickness value, derived from step S4. This indicates outliers in the normalized aircraft surface image, originating from step S5. The outliers in the normalized ground image are derived from step S6; S8. Determine whether the fusion value is greater than the preset threshold. If the fusion value is less than or equal to the preset threshold, there is no frost in the current detection area on the aircraft surface. Then return to step S4 to perform the next round of detection on the aircraft surface. If the fusion value is greater than the preset threshold, there is frost in the current detection area on the aircraft surface. The detection system will trigger a defrosting alarm and perform a defrosting process.

[0026] In a preferred embodiment, the laser reflection acquisition in step S4, the aircraft surface image acquisition in step S5, and the ground area image acquisition in step S6 are performed simultaneously and recorded with the same timestamp in the detection system.

[0027] Example 1 S1. Set up an aircraft surface frost detection system and configure the system for training operation mode. In this embodiment, the system is set up on an outdoor apron at an airport to detect frost on the surface of a parked civil aircraft. The system consists of multiple sub-modules, all integrated on an integrated detection bracket, which is installed approximately 1.2 meters from the leading edge of the wing and is used to fix the laser, image acquisition device, and environmental sensors.

[0028] Used to emit wavelength The laser beam is mounted on top of the detection bracket and angled towards the leading edge of the wing so that the emitted laser forms an incident angle with the wing surface. In this embodiment, Set to 532nm, the laser is controlled by an independent power supply module, with a fixed output power, and the corresponding initial light intensity is denoted as [insert value here]. The center height of the laser head is approximately 1.4 meters to ensure that the laser spot falls stably on the detection area.

[0029] A monochrome industrial camera for receiving laser reflection signals is mounted below the laser. The imaging optical axis forms a non-collinear angle with the laser incident direction. A center wavelength is mounted in front of the lens. A narrowband filter, for example, with a bandwidth of ±10nm, is used to shield ambient background light. The camera resolution is set to 1280×1024 and connected to the main control computing module.

[0030] An RGB camera for acquiring color images is mounted on the side of the support, with its lens facing the leading edge of the aircraft, covering the surface of the target and the ground area below it. The camera has a resolution of 2048×2048, a fixed focal length lens, and the acquired images are used for subsequent visual anomaly detection and ground anomaly detection.

[0031] A set of LED auxiliary light source modules is installed below the camera, evenly distributed above the detection area, to enhance image brightness in nighttime or low-light conditions. All imaging modules are connected to the embedded edge computing host via a USB interface.

[0032] The environmental parameter acquisition module includes an integrated temperature and humidity sensor, which is mounted on the lower part of the bracket at the same height as the wing edge, and can simultaneously collect the current air temperature. With relative humidity This is used for subsequent environmental state estimation.

[0033] All sensor modules are connected to the edge computing unit via a unified power supply and data bus. The main control computing module comes pre-installed with control software, supporting the automatic startup of the main detection program after system power-on and completing data link testing and hardware status verification. After the system completes its power-on self-test, it has the physical basis to enter the operating mode.

[0034] S2. Conduct offline training on the detection system, collect aircraft surface image samples under frost-free and normal environmental conditions, and construct an image feature benchmark library for aircraft surface under frost-free conditions. In this embodiment, before formal testing, the system first constructs an image feature library under frost-free conditions through on-site data acquisition. The acquisition location is the airport parking area, and the target object is a similar type of civil aircraft. Under conditions of two consecutive days without precipitation and ground temperature above the frost point temperature, the system switches to training mode and uses a color industrial camera to acquire images of the aircraft surface. The image resolution is set to 2048×2048, the exposure time is 100 ms, and 20 frames of images are acquired in each time period, covering typical lighting conditions.

[0035] The system divides each image into several image blocks, each image block being denoted as... and use the loaded convolutional neural network model for each Calculate the corresponding eigenvectors This feature vector is used to characterize the texture, brightness, and structure distribution of the region in a frost-free state. The feature vectors of all image patches are numbered according to their location in the original image and organized into a standard feature set, denoted as... Each of them This corresponds to an image patch that has been confirmed as a frost-free area.

[0036] Once the feature library is built, it is saved to a local path in binary format and permanently loaded and used in the system's detection mode, without being updated. During subsequent detection processes, all image outliers will be compared against this feature library.

[0037] This feature library contains only image representations under normal operating conditions and serves as an important reference benchmark for subsequently determining the degree of abnormal offset in image regions.

[0038] S3. The detection system is put into operation. The system switches its operating mode to detection mode and performs initialization settings. In this embodiment, the system needs to complete the measurement of environmental parameters and the calibration of the optical properties of the frost layer before entering the detection process, in order to support the subsequent thickness inversion model and visual judgment algorithm.

[0039] First, the system collects the current environmental conditions using temperature and humidity sensors mounted on the detection bracket. The detection took place at 23:00 on January 2nd, and the system recorded the ambient temperature. relative humidity All measurement data is stored in real-time in text format on a local path.

[0040] Based on the established model, the system uses the following empirical formula to estimate the frost point temperature for that evening. : ; in, , , , which are the empirical coefficients obtained through experimental fitting in this embodiment. After substituting the data, the system calculates... This data serves as the environmental basis for subsequent frost formation determination. After completing the environmental parameter acquisition, the system calibrates the optical properties of the frost layer, including the scattering coefficient. With pore structure correction factor The determination.

[0041] To determine the scattering coefficient The operator selected a frosted plate with a known thickness D=2.0 mm in the test area, started laser irradiation, and collected the reflected light intensity. After filtering and background subtraction, the light intensity is compared with the initial light intensity. Comparison, substituting into the Beer–Lambert law: ; Taking the logarithmic transformation, we can calculate... This value is used for the exponential decay term in subsequent thickness inversion.

[0042] Meanwhile, to correct for the non-uniform scattering caused by the presence of micropores within the frost, the system uses previous experimental statistical results to set a porosity correction coefficient. This value and It is stored together in the main control program's configuration file and automatically loaded in subsequent model calls.

[0043] At this point, the system has completed the environmental measurements and optical parameter calibration before testing, and is ready to enter the formal data acquisition process.

[0044] S4. Calculate the frost thickness in the target detection area using the laser reflection principle and light intensity attenuation model. In this embodiment, the system uses oblique incidence laser illumination and reflection image acquisition to estimate the thickness of frost on the aircraft surface. The laser emission wavelength is set to... The angle of incidence is The initial laser intensity calibration value is unit.

[0045] A monochrome industrial camera is mounted to one side of the laser, with its imaging direction forming an angle with the laser direction. A bandpass filter is fixed in front of the lens to isolate background light. Five frames of reflection images are captured in each round of detection. After filtering, the average brightness of the target area is extracted as the measured reflected light intensity. .

[0046] According to Beer–Lambert's law, the relationship between laser transmitted light intensity and thickness is as follows: ; in, This indicates the intensity of transmitted light behind the frost layer. The scattering coefficient is... This refers to the thickness of the frost layer.

[0047] The intensity of reflected light caused by frost can be expressed as: ; When satisfied When conditions are met, the exponential term can be expanded as follows: ; Substituting into the above equation, we obtain an approximate relationship: ; Therefore, thickness It can be determined by the measured reflected light intensity The approximate inversion is as follows: ; In this embodiment, the measured reflected light intensity is initial intensity of laser scattering coefficient Substituting, we get: ; The system repeats the above inversion process five times and calculates the average value to finally obtain the frost layer thickness value. This is the laser thickness output for the current detection frame.

[0048] S5. Acquire images of the aircraft surface in the current detection area using a color camera, further extract image features, and compare them with the image feature benchmark library of the aircraft surface under frost-free conditions constructed in step S2 to calculate outliers in the current detection area. In this embodiment, the system acquires images of the aircraft surface using a color industrial camera and assesses the degree of anomaly based on the characteristics of the image patches compared with standard samples. The image resolution is 2048×2048, and the acquisition time is synchronized with laser irradiation.

[0049] The system divides the image into multiple image blocks, denoted as... Extracting its feature vectors through a convolutional neural network The feature vector has a dimension of 512, representing the texture and structure distribution of a local region. The system also utilizes the frost-free feature library built during the training phase. And calculate the following outliers: ; In this embodiment, a certain image block The Euclidean distance between the extracted features and the matching samples is calculated as follows: .

[0050] The system iterates through all image patches and performs multi-scale aggregation to calculate outliers across the entire image. ; ; Set two scales and The mean outliers were 0.21 and 0.31, respectively, corresponding to weights... , The calculation yields: ; This value is used for subsequent fusion judgments and indicates the overall degree of visual abnormality in the current image frame.

[0051] S6. Acquire images of the ground area using a color camera and calculate the outliers in the ground area. In this embodiment, the system synchronously acquires ground area images during each round of detection and uses them to assess potential abnormal interference introduced by changes in the overall environment. The images are acquired by a fixed-angle color camera with a resolution of 2048×2048, and the acquisition time is the same as that of the main camera.

[0052] The system inputs ground images into a lightweight convolutional neural network model, performs end-to-end scoring, and outputs ground area outliers. In the current detection frame, the system model calculation result is as follows: ; This outlier is not included in the frost thickness calculation, but is only used as an image-level auxiliary reference to improve the system's robustness to background changes such as environmental occlusion and sudden changes in illumination, and is used as one of the weighted channels in subsequent fusion decisions.

[0053] S7. Perform data normalization on the frost layer thickness calculated in step S4, the aircraft surface anomaly value calculated in step S5, and the ground area anomaly value calculated in step S6. Then, perform weighted fusion on the normalization results to calculate the final fused value. In this embodiment, the system will use laser thickness Image outliers and ground area outliers After normalization and standardization, the weighted averages are combined to form a comprehensive index. .

[0054] The normalization method is as follows: The possible values ​​are as follows: ; ; ; The normalization result is: ; ; ; Set the weighting coefficient as , , The fusion value is calculated as follows: ; ; The system will display the results. Write to the cache for subsequent judgment.

[0055] S8. Determine if the fusion value is greater than the preset threshold. If the fusion value is less than or equal to the preset threshold, there is no frost in the current detection area on the aircraft surface. Then return to step S4 to perform the next round of detection on the aircraft surface. If the fusion value is greater than the preset threshold, there is frost in the current detection area on the aircraft surface. The detection system will trigger a defrosting alarm and perform defrosting. System comparison fusion value With threshold Determine if there is a risk of frost formation.

[0056] Current fusion value:; Threshold setting: ; because The system detects the presence of frost and triggers the alarm module. The alarm module writes the alarm record to the log, which includes: 1) Current frame number; 2) , and ; 3) , and ; 4) And the judgment result; 5) Timestamp; The system also performs trend judgment on continuous detection results. If three consecutive frames... If the temperature is below the threshold for consecutive frames, it is considered a persistent frosting trend, and the alarm will remain in effect; The system will then deactivate the alarm.

[0057] S9. Closed-loop control and continuous operation mechanism The system adopts a closed-loop operation mode, entering a continuous acquisition-analysis-judgment process after initialization. Each detection cycle is 5 minutes, which is started and completed automatically by the main control program at regular intervals.

[0058] After each round of testing, the system will store the following data in a local record table: , and Normalization results and Current alarm status.

[0059] After the initial testing phase is completed, the system immediately begins the next round of testing. The front-end interface displays the results in real time. The change curves and alarm status support 24 / 7 operation without manual intervention.

[0060] Example 2 The following will illustrate the actual operational performance of the system described in this invention using a specific example of frost detection on a parked civil aircraft. This embodiment is based on... Figure 2 The multimodal fusion architecture shown was tested on an aircraft parked at an airport, and the system completed the entire process from initialization to closed-loop determination.

[0061] S1. Set up an aircraft surface frost detection system and configure the system for training operation mode. The system architecture in this embodiment is as follows: Figure 3 As shown, the laser, monochrome camera, color camera, and temperature and humidity sensor are mounted on the system bracket, respectively, and aimed at the leading edge of the wing and the ground area. Each sensor is connected to the main computer via high-speed data cables and is configured with synchronous trigger signals for control. The system selects "detection mode" upon startup and enters automatic operation.

[0062] S2. Conduct offline training on the detection system, collect aircraft surface image samples under frost-free and normal environmental conditions, and construct an image feature benchmark library for aircraft surfaces under frost-free conditions. The system acquires images of the wing area under frost-free conditions, extracts features from image patches using a convolutional neural network, and constructs a standard frost-free feature library. This feature library is stored in a database as a comparison reference for subsequent detection.

[0063] S3. The detection system is put into operation. The system switches its operating mode to detection mode and performs initialization settings. Laser wavelength set to The incident angle is set to Initial laser intensity Units. The color camera resolution was set to 2048×2048 pixels, the exposure time was 120 milliseconds, and the data acquisition period was from 23:00 on the current day to 06:00 the next day. The ambient temperature was... relative humidity Calculate the frost point temperature Based on experimental calibration, the system scattering coefficient was set. Pore ​​correction coefficient The detection area was selected as the leading edge of the wing.

[0064] S4. Calculate the frost thickness in the target detection area using the laser reflection principle and light intensity attenuation model. A laser beam is obliquely irradiated onto the leading edge area of ​​the wing. A monochrome camera captures the image, and the average brightness is extracted after processing with a filter. According to the light intensity attenuation model, the frost layer thickness... The calculation is as follows: ; The system records this value as the laser channel output. .

[0065] S5. Acquire images of the aircraft surface in the current detection area using a color camera, such as... Figure 8 The image shown is an RGB image acquired in this embodiment. Its image features are further extracted and compared with the image feature benchmark library of the aircraft surface under frost-free conditions constructed in step S2. Outliers in the current detection area are calculated. The specific matching and memory library construction process is as follows: Figure 4 As shown.

[0066] The system synchronously acquires images of the aircraft surface and divides them into multiple image blocks. Feature vectors are extracted from each block. Features in the frost-free feature library Perform comparisons and calculate outliers. Multi-scale aggregation yields outliers across the entire image: ; The image exhibits features of enhanced diffuse reflection and structural distortion, indicating the presence of a significant anomaly.

[0067] S6. Acquire images of the ground area using a color camera and calculate the outliers in the ground area. Ground area images are captured by a separate camera, input into a lightweight convolutional neural network model to evaluate the overall image state, and the system outputs ground area outliers. ; This outlier serves as auxiliary information input for the image channels, used in multimodal fusion.

[0068] S7. Perform data normalization on the frost layer thickness calculated in step S4, the aircraft surface anomaly value calculated in step S5, and the ground area anomaly value calculated in step S6. Then, perform weighted fusion on the normalization results to calculate the final fused value. The fusion structure in this embodiment is as follows: Figure 5 As shown.

[0069] The system normalizes the three channel outputs separately: ; ; ; The weighting coefficient is , , Calculate the fusion value: ; like Figure 10 As shown, after introducing a multi-source data fusion processing mechanism, the frost thickness value obtained from the detection results is significantly more stable and closer to the baseline value. Figure 11 As shown, multi-source data fusion effectively reduces the cross-entropy between the detection results and the benchmark value.

[0070] S8. Determine whether the fusion value is greater than the preset threshold. If the fusion value is less than or equal to the preset threshold, there is no frost in the current detection area on the aircraft surface. Then return to step S4 to perform the next round of detection on the aircraft surface. If the fusion value is greater than the preset threshold, there is frost in the current detection area on the aircraft surface. The detection system will trigger a defrosting alarm and perform defrosting. The system sets risk thresholds. .because The system detects the presence of frost, immediately triggers an alarm signal, and records the data for this round to the log. (This is an example.) Figure 6 This curve represents the change in the degree of frost formation during the operating period. Figure 7 This represents the curve showing the change in the measured frosted area over time.

[0071] S9. Closed-loop control and continuous operation mechanism The system automatically performs a complete test every 5 minutes for three consecutive rounds. Given values ​​of 0.936, 0.911, and 0.922 respectively, calculate the moving average: ; The system determines that the frost layer persists, maintains the alarm output, and proceeds to the next round of detection, operating in a closed loop without manual intervention. Simultaneously, the visualization results of the detected frost values ​​in this embodiment are as follows: Figure 9 As shown.

[0072] This invention addresses the need for frost detection on civil aircraft surfaces. It constructs a multimodal fusion scheme based on a laser scattering model and a visual image anomaly detection algorithm, fully considering the complex mechanism and optical characteristics of frost formation. According to the modified Beer-Lambert law, a linear relationship is established between frost thickness and diffuse reflection light intensity, enabling real-time frost thickness detection. A multi-scale visual feature extraction algorithm based on a convolutional neural network can quickly identify large-scale frost coverage and localized icing anomalies. By linearly weighting and fusing these two types of detection information, frost monitoring is achieved. This system exhibits higher robustness to environmental interference and can promptly trigger alarms and de-icing operations, effectively reducing the burden on aircraft maintenance personnel while enhancing the safety and efficiency of aircraft ground operations. The system maintains good stability even in apron environments with large temperature fluctuations and variable lighting, demonstrating its potential for widespread application in rapid ground detection before and after aircraft parking or flight, and providing support for smart airport construction.

[0073] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for detecting frost formation on aircraft surfaces based on laser thickness measurement and visual image analysis, characterized in that: It includes the following steps: S1. Set up an aircraft surface frost detection system and configure the system for training operation mode. S2. Conduct offline training on the detection system, collect aircraft surface image samples under frost-free and normal environmental conditions, and construct an image feature benchmark library for aircraft surface under frost-free conditions. S3. Set the detection system to detection mode and initialize the detection system. S4. Calculate the frost thickness in the target detection area using the laser reflection principle and light intensity attenuation model; Step S4 is as follows: S41. Start the laser, and the laser will irradiate the target detection area at the set incident angle; S42. After the black and white camera receives the reflected signal from the surface of the aircraft, it processes the signal through a filter to obtain an effective image signal in a single band. S43. Extract the light intensity value corresponding to the effective image in step S42, that is, the reflected light intensity actually collected by the black and white camera. S44. Calculate the intensity of the transmitted light after the laser passes through the frost layer. The expression is: ； In the formula, This represents the intensity of transmitted light under ideal conditions. Indicates the intensity of the incident light. Represents an exponential function. Indicates the scattering coefficient of the frost layer. Indicates the thickness of the frost layer; S45. Define the diffuse reflectance of laser light by a frost layer, and its expression is: ； S46. Construct a linear approximation model under thin-layer conditions, that is, in Under the given conditions, the linear approximation model is: ； In the formula, express higher-order infinitesimals; S47. Substitute the actual reflected light intensity collected in step S43 into the linear approximation model to calculate the frost layer thickness, the expression of which is: ； In the formula, This represents the intensity of the reflected light obtained from actual measurement; S48. Obtain multiple laser measurement values ​​and calculate the average thickness of the frost layer; S5. Acquire images of the aircraft surface in the current detection area using a color camera, extract image features, and compare them with the image feature benchmark library of the aircraft surface under frost-free conditions constructed in step S2 to calculate the outlier value in the current detection area. S6. Acquire images of the ground area using a color camera and calculate outliers in the ground area; S7. Perform data normalization on the frost layer thickness calculated in step S4, the aircraft surface anomaly value calculated in step S5, and the ground area anomaly value calculated in step S6. Then, perform weighted fusion on the normalization results and calculate the final fused value. The calculation formula is as follows: ； In the formula, This indicates the fusion weight of the laser thickness measurement channel. The fusion weights represent outliers in aircraft surface images. The fusion weights represent outliers in the ground region. This represents the normalized laser-measured frost thickness value. This represents outliers in the normalized aircraft surface image. Indicates outliers in the normalized ground image; S8. Determine whether the fusion value is greater than the preset threshold. If the fusion value is less than or equal to the preset threshold, there is no frost in the current detection area on the aircraft surface. Then return to step S4 to perform the next round of detection on the aircraft surface. If the fusion value is greater than the preset threshold, there is frost in the current detection area on the aircraft surface. The detection system will trigger a defrosting warning and perform defrosting.

2. The method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: The laser reflection acquisition in step S4, the aircraft surface image acquisition in step S5, and the ground area image acquisition in step S6 are performed simultaneously and recorded with the same timestamp in the detection system.

3. The method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: The specific steps of step S1 are as follows: S11. Arrange the laser, black and white camera, color camera and temperature and humidity sensor on the detection system bracket respectively, and place the detection system bracket on one side of the aircraft wing. S12. Adjust the incident angle of the laser on the aircraft surface; S13. Adjust the field of view of the black-and-white camera and the color camera on the aircraft surface and the ground area. S14. Connect the laser, black and white camera, color camera and temperature and humidity sensor to the remote server of the detection system respectively. S15. Configure a time synchronization mechanism in the detection system so that laser measurement, image acquisition and environmental detection parameter acquisition can be completed synchronously in each acquisition cycle; S16. Test the data integrity of each data transmission channel and set the operating mode of the detection system to training mode.

4. The method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: The specific steps of step S2 are as follows: S21. Acquire images of the aircraft surface under normal environmental conditions without ice or frost. S22. Repeat step S21 under different time periods and different lighting conditions to cover the range of typical visual changes on the aircraft surface and ground area. S23. Mark all image data on the aircraft surface and ground area collected in step S22 as being in an anomaly-free state and use them as training samples. S24. Use a preset image feature extraction algorithm to encode each acquired image; S25. Divide each image into multiple image blocks and use a pre-trained convolutional neural network model to calculate the feature vectors corresponding to different image blocks; S26. Store the feature vectors of all image blocks into the image feature reference library in regional order, and maintain the correspondence with the original image.

5. The method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: Step S3 is as follows: S31. Set the optical parameters of the laser; S32, Set the resolution, exposure time, and brightness of the auxiliary light source for the black-and-white camera and the color camera; S33. Select a fixed detection area on the aircraft surface and set the scattering coefficient, porosity correction coefficient, and frost detection threshold.

6. The method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: Step S5 is as follows: S51. Acquire color images of the aircraft surface in the current detection area using a color camera; S52. Divide the color image of the aircraft surface acquired in step S51 into multiple image blocks, and use a pre-trained convolutional neural network model to calculate the feature vectors corresponding to different image blocks. S53. Call the image feature benchmark library of the aircraft surface under frost-free conditions built during offline training; S54. Calculate the outlier values ​​on the aircraft surface in the current detection area.

7. The method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis according to claim 6, characterized in that: Step S54 is as follows: S541. Calculate the minimum Euclidean distance between the feature vector of each image patch in the acquired color image of the aircraft surface and the feature vector of the image under frost-free conditions. The expression is as follows: ； In the formula, The feature vector representing an image patch. This represents the image feature vector in a frost-free state. S542. The nearest neighbor feature matching and outlier calculation method is used to perform feature matching calculation on the entire aircraft surface image acquired. S543. Aggregate the outliers of all image patches and calculate the outliers of the entire image. The expression is as follows: ； In the formula, This represents the outlier results for the entire surface of the aircraft. This represents the set of image scales participating in the aggregation. This indicates the current scale level under consideration. Indicated in scale The total number of image blocks divided into subdivisions, Indicates the first Image content of each image block Representing scale Next, the Outliers in an image patch Representing scale The average outlier value of all image patches below. Representing scale Weighting coefficients; S544. Record and store the abnormal values ​​on the surface of the aircraft in the current detection area.

8. The method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: Step S6 is as follows: S61. Acquire images of the ground area using a color camera; S62. The collected ground images are directly input into the convolutional neural network model; S63. The model performs an overall analysis of the input image and outputs outliers in the ground area.

Images