Aircraft surface frosting detection method based on laser thickness measurement and visual image analysis

Abstract

The invention provides an airplane surface frosting detection method based on laser thickness measurement and visual image analysis. The airplane surface frosting detection method comprises the following steps: S1, building an airplane surface frosting detection system; s2, training and constructing an image feature reference library in a frost-free state; s3, performing initialization setting on the detection system; s4, calculating the thickness of a frost layer; s5, acquiring an aircraft surface image and calculating an abnormal value; s6, collecting an image of the ground area and calculating an abnormal value; s7, performing normalization and weighted fusion calculation on the multi-source data; and S8, whether the fusion value is larger than a preset threshold value or not is judged, if the fusion value is larger than the preset threshold value, it is judged that frost exists in the current detection area on the surface of the aircraft, and the detection system triggers a defrosting prompt alarm and conducts defrosting. According to the method, a multi-modal fusion scheme is constructed based on the laser scattering model and the visual image anomaly detection algorithm, the complex mechanism and optical characteristics of frost layer formation are considered, the method has higher robustness to environmental interference, alarm and deicing operation can be triggered in time, the burden of aircraft maintenance personnel is effectively reduced, and the aircraft maintenance efficiency is improved. And the ground operation safety and efficiency of the aircraft are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aviation detection, in particular to a method for detecting frost on the surface of an airplane based on laser thickness measurement and visual image analysis. BACKGROUND

[0002] Frost on the surface of an airplane is an important safety issue that has long been concerned in the field of civil aviation. In particular, in low-temperature, high-humidity or snowfall environments, the deposition of ice and frost on the surface of the fuselage can affect the aerodynamic characteristics of the airplane, increase the flight resistance, reduce the lift, and in severe cases, can cause control failure or take-off obstruction. Therefore, civil aviation authorities of various countries have put forward strict requirements for the absence of frost on the surface of the airplane before take-off, and the airplane should ensure that there is no frost attached to the key parts before take-off.

[0003] Currently, the detection of frost on the surface of an airplane mainly relies on manual visual inspection and traditional meteorological data speculation. The manual inspection method relies on experience and is inefficient, susceptible to subjective errors, and cannot achieve real-time monitoring. The prediction method based on meteorological data can provide certain frost formation trend, but cannot directly reflect the actual thickness and distribution of frost on the surface of the airplane.

[0004] With the increasing demand for detection of frost on the surface of an airplane, a large number of researches have focused on single modal sensor detection, such as deep learning algorithm based on visible light image, temperature anomaly analysis based on infrared radiation, etc. However, these methods generally have the problems of insufficient generalization ability, low environmental adaptability and poor real-time performance, which are difficult to meet the needs of efficient scheduling and intelligent operation of the airport. Therefore, how to integrate multiple sensing technologies, break through the limitations of single sensor, and realize high-precision and high-robustness frost detection technology has become an important research direction of aviation safety guarantee. SUMMARY

[0005] In view of the problems existing in the prior art, the present application provides a method for detecting frost on the surface of an airplane based on laser thickness measurement and visual image analysis. In view of the demand for detecting frost on the surface of a civil airplane, a multi-modal fusion scheme is constructed based on a laser scattering model and a visual image anomaly detection algorithm, fully considering the complex mechanism and optical properties of frost layer formation, and has the potential to be widely applied to rapid ground detection before and after the civil airplane stops or flies, providing support for the construction of smart airport.

[0006] In one aspect, the present application provides a method for detecting frost on the surface of an airplane based on laser thickness measurement and visual image analysis, which comprises the following steps: S1, a system for detecting frost on the surface of an airplane is built, and the detection system is set to a training operation mode; S2, the detection system is trained offline, image samples of the surface of the airplane under the conditions of no frost and normal environment are collected, and an image feature reference library under the condition of no frost on the surface of the airplane is constructed; S3. The detection system is set to detection mode and the detection system is initialized; S4. Calculate the frost thickness in the target detection area using the laser reflection principle and light intensity attenuation model; S5. Capture an image of the aircraft surface in the current detection area using a color camera, extract image features, and compare the image features with the image feature reference library of the aircraft surface in a frost-free state constructed in step S2 to calculate an outlier value in the current detection area. S6. collecting an image of the ground area using a color camera and calculating an outlier value of the ground area; S7, performing data normalization processing on the frost layer thickness calculated in step S4, the aircraft surface anomaly calculated in step S5, and the ground area anomaly calculated in step S6, and performing weighted fusion on the normalized processing results to calculate the final fusion value, which is calculated as follows: ; Where, represents the fusion weight of the laser thickness measurement channel, represents the fusion weight of the outliers in the aircraft surface image, represents the fusion weight of the outliers in the ground area, represents the normalized frost thickness value measured by laser, represents the normalized aircraft surface image outlier value, represents the normalized ground image outlier value; S8. Determine whether the fusion value is greater than a preset threshold. If the fusion value is less than or equal to the preset threshold, then there is no frost in the current detection area on the aircraft surface, and the process returns to step S4 to perform the next round of detection on the aircraft surface. If the fusion value is greater than the preset threshold, then there is frost in the current detection area on the aircraft surface, and the detection system triggers a defrost warning alarm and performs defrosting.

[0007] Preferably, the laser reflection acquisition in step S4, the aircraft surface image acquisition in step S5, and the ground area image acquisition in step S6 are performed simultaneously, and the same timestamp is recorded in the detection system.

[0008] Furthermore, the specific steps of step S1 are: S11, arranging the laser, black and white camera, color camera, and temperature and humidity sensor on a detection system bracket, and placing the detection system bracket on one side of the aircraft wing; S12, adjusting the incident angle of the laser on the aircraft surface; S13, adjusting the field of view of the black and white camera and the color camera on the aircraft surface and the ground area; S14, connect the laser, black and white camera, color camera and temperature and humidity sensor with the remote server of the detection system respectively; S15, configure a time synchronization mechanism in the detection system, which can complete the laser measurement, image acquisition and environmental detection parameter acquisition synchronously in each round of acquisition period; S16, test the data integrity of each data transmission channel, and set the running mode of the detection system to the training running mode.

[0009] Further, the specific steps of step S2 are: S21, image acquisition is performed on the aircraft surface under non-icing, normal environmental conditions; S22, repeat step S21 under different time periods and different light conditions to cover the typical visual changes on the aircraft surface and the ground area; S23, mark all image data collected on the aircraft surface and the ground area in step S22 as a non-abnormal state, and use it as a training sample; S24, use a pre-set image feature extraction algorithm to encode each collected image; S25, divide each image into multiple image blocks, and calculate the feature vectors corresponding to different image blocks using a pre-trained convolutional neural network model; S26, store the feature vectors of all image blocks in the image feature reference library in order of area, and maintain the corresponding relationship with the original image.

[0010] Further, the specific steps of step S3 are: S31, set the optical parameters of the laser; S32, set the resolution, exposure time of the black and white camera and color camera, and the brightness of the auxiliary light source; S33, select a fixed detection area on the aircraft surface, and set the scattering coefficient, porosity correction coefficient and frost determination threshold.

[0011] Further, the specific steps of step S4 are: S41, start the laser, and the laser is irradiated to the target detection area according to the set incident angle; S42, after the black and white camera receives the reflection signal of the aircraft surface, a single waveband effective image signal is obtained through optical filter processing; S43, extract the light intensity value corresponding to the effective image in step S42, i.e. the actual reflection light intensity collected by the black and white camera; S44, calculate the transmission light intensity of the laser after passing through the frost layer, which is expressed as: ; In the formula, represents the intensity of the transmitted light in the ideal case, represents the intensity of the incident light, represents the exponential function, represents the scattering coefficient of the frost layer, represents the thickness of the frost layer; S45, defines the amount of diffuse reflection of the laser by the frost layer, which is expressed as: ; S46, constructs a linear approximation model under the thin layer condition, that is, under the condition that the linear approximation model is: ; wherein, represents a high-order infinitesimal; S47, substitutes the actual reflected light intensity collected in step S43 into the linear approximation model to calculate the thickness of the frost layer, which is expressed as: ; wherein, represents the actual measured reflected light intensity; S48, obtains multiple laser measurement values and calculates the average value of the thickness of the frost layer.

[0012] Preferably, step S5 comprises the following steps: S51, collects a color image of the aircraft surface in the current detection area by a color camera; S52, divides the color image of the aircraft surface collected in step S51 into multiple image blocks, and calculates the feature vectors corresponding to different image blocks by using a pre-trained convolutional neural network model; S53, calls an image feature benchmark library under the ice-free state of the aircraft surface constructed during offline training; S54, calculates the abnormal value of the aircraft surface in the current detection area.

[0013] Further, step S54 comprises the following steps: S541, calculates the minimum Euclidean distance between the feature vector of each image block in the collected color image of the aircraft surface and the image feature vector under the ice-free state, which is expressed as: ; wherein, represents the feature vector of the image block, represents the image feature vector under the ice-free state; S542, adopts the nearest neighbor feature matching and abnormal value calculation method to perform feature matching calculation on the collected whole aircraft surface image. S543, aggregate the outliers of all image blocks, and calculate the outliers of the whole image, the expression is: ; In the formula, represents the whole image outlier result of the whole aircraft surface, represents the image scale set participating in aggregation (such as different resolution levels), represents a certain scale level considered at present, represents the total number of image blocks divided under the scale represents the image content of the th image block, represents the outlier of the th image block under the scale represents the average outlier of all image blocks under the scale represents the weight coefficient of the scale ; S544, record and store the outliers of the aircraft surface of the current detection area.

[0014] Further, the specific steps of step S6 are: S61, acquiring the ground area image through a color camera; S62, directly inputting the acquired ground image into a convolutional neural network model; S63, the model performs overall analysis on the input image, and outputs the outliers of the ground area.

[0015] Compared with the prior art, the beneficial effects of the present application are as follows: 1. The aircraft surface frost detection method based on laser thickness measurement and visual image analysis provided by the present application constructs a multi-modal fusion scheme for the demand of civil aircraft surface frost detection, fully considers the complex mechanism and optical properties of frost layer formation, and according to the modified Beer-Lambert law, establishes a linear relationship between frost layer thickness and diffuse reflection light intensity, realizes real-time frost layer thickness detection, and the multi-scale visual feature extraction algorithm based on convolutional neural network can quickly identify large-scale frost layer coverage and local icing abnormalities.

[0016] 2. The aircraft surface frost detection method based on laser thickness measurement and visual image analysis provided by the present application realizes frost layer monitoring through linear weighted fusion of frost layer thickness and diffuse reflection light intensity two kinds of detection information, the system has higher robustness to environmental interference, and can timely trigger alarm and deicing operation, thereby effectively reducing more human judgment in aircraft maintenance process, and enhancing the safety and efficiency of aircraft ground operation.

[0017] ​​​3、The aircraft surface frost detection method based on laser thickness measurement and visual image analysis provided by the present application can maintain good stability in the airport environment where temperature fluctuates in a large range and light changes, has the potential to be widely applied to the ground rapid detection of civil aircraft before and after parking or flight, and can provide support for the construction of smart airports. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 It is a flowchart of the aircraft surface frost detection method based on laser thickness measurement and visual image analysis of the present application; Figure 2 It is a flowchart of the aircraft surface frost detection method of embodiment 1 of the present application; Figure 3 It is a system structure diagram of the aircraft surface frost detection method of embodiment 1 of the present application; Figure 4 It is a visual outlier algorithm structure diagram of embodiment 2 of the present application; Figure 5 It is a fusion decision algorithm structure diagram of embodiment 2 of the present application; Figure 6 It is a frost layer thickness data diagram measured by embodiment 2 of the present application; Figure 7 It is a frost area proportion data diagram measured by embodiment 2 of the present application; Figure 8 It is an RGB image collected by embodiment 2 of the present application; Figure 9 It is a visualization rendering result diagram of frost thickness measured by embodiment 2 of the present application; Figure 10 It is a comparison line chart of single radar modal measurement, single visual modal measurement and fusion measurement; Figure 11 It is a histogram diagram of ablation mean square error. DETAILED DESCRIPTION

[0019] The aircraft surface frost detection method based on laser thickness measurement and visual image analysis of the present application, as shown in Figure 1 、 2 and Figure 3 , comprises the following steps: S1, build an aircraft surface frost detection system, set the detection system to a training operation mode, and the specific steps are: S11, arrange the laser, black and white camera, color camera and temperature and humidity sensor on the detection system support respectively, and place the detection system support on one side of the aircraft wing; S12, adjust the incident angle of the laser on the aircraft surface; S13, adjust the field of view range of the black and white camera and the color camera on the aircraft surface and the ground area; S14, connect the laser, black and white camera, color camera and temperature and humidity sensor with the remote server of the detection system respectively; S15, configure a time synchronization mechanism in the detection system, which can complete the laser measurement, image acquisition and environmental detection parameter acquisition synchronously in each round of acquisition period; S16, test the data integrity of each data transmission channel, and set the running mode of the detection system to the training running mode.

[0020] S2, offline training of the detection system, collecting aircraft surface image samples under ice-free, normal environmental conditions, and constructing an image feature reference library under the ice-free state of the aircraft surface, wherein ice-free means that the ice coverage is zero, and the specific steps are: S21, image acquisition of the aircraft surface under ice-free, normal environmental conditions; S22, repeat step S21 under different time periods and different light conditions to cover the typical visual changes on the aircraft surface and the ground area; S23, mark all image data collected on the aircraft surface and the ground area in step S22 as normal state, and use it as a training sample; S24, use a pre-set image feature extraction algorithm to encode each collected image; S25, divide each image into multiple image blocks, and calculate the feature vectors corresponding to different image blocks using a pre-trained convolutional neural network model; S26, store the feature vectors of all image blocks in the image feature reference library in order of area, and maintain the corresponding relationship with the original image.

[0021] S3, the detection system is put into operation, the running mode of the detection system is switched to the detection mode, and the detection system is initialized, and the specific steps are: S31, set the optical parameters of the laser; S32, set the resolution, exposure time of the black and white camera and color camera, and the brightness of the auxiliary light source; S33, select a fixed detection area on the aircraft surface, and set the scattering coefficient, porosity correction coefficient and frost determination threshold.

[0022] S4, calculate the frost layer thickness of the target detection area using the laser reflection principle and the light intensity attenuation model, and the specific steps are: S41, start the laser, and the laser is irradiated to the target detection area according to the set incident angle; S42, after the black and white camera receives the reflection signal of the aircraft surface, a single waveband effective image signal is obtained through the optical filter processing; S43, extract the light intensity value corresponding to the effective image in step S42, that is, the actual reflected light intensity collected by the black and white camera; S44, according to Beer-Lambert law, calculate the transmitted light intensity after the laser passes through the frost layer, which is expressed as: ; In the formula, represents the transmitted light intensity in the ideal case, represents the incident light intensity, represents the exponential function, represents the scattering coefficient of the frost layer, represents the thickness of the frost layer; S45, define the diffuse reflection amount of the frost layer to the laser, which is expressed as: ; S46, construct a linear approximation model under thin layer conditions, that is, under the condition that the linear approximation model is: ; In the formula, represents high-order infinitesimal; S47, substitute the actual reflected light intensity collected in step S43 into the linear approximation model to calculate the thickness of the frost layer, which is expressed as: ; In the formula, represents the actual measured reflected light intensity; S48, obtain multiple laser measurement values and calculate the average value of the thickness of the frost layer.

[0023] S5, collect the aircraft surface image of the current detection area by the color camera, further extract the image features, and compare with the image feature reference library of the aircraft surface without frost in step S2 to calculate the abnormal value of the current detection area, the specific steps are as follows: S51, collect the aircraft surface color image of the current detection area by the color camera; S52, divide the aircraft surface color image collected in step S51 into multiple image blocks, and calculate the feature vectors corresponding to different image blocks by using the pre-trained convolutional neural network model; S53, call the image feature reference library of the aircraft surface without frost in the offline training; S54, calculate the abnormal value of the aircraft surface of the current detection area, the specific steps are as follows: S541, calculate the minimum Euclidean distance between the feature vector of each image block in the collected aircraft surface color image and the image feature vector under the no-frost state, which is expressed as: ; wherein, represents the feature vector of the image block, represents the image feature vector in the frost-free state; S542, using the nearest neighbor feature matching and outlier calculation method, the feature matching calculation is carried out on the collected whole aircraft surface image, and the expression is: ; wherein, represents the current collected aircraft surface image, represents the set after the image is divided into multiple image blocks, is the feature vector extracted from a certain image block, which is used to represent the texture and structure information of the local area, is a certain reference feature vector in the normal image feature library established in the frost-free state, which represents the image feature distribution in the ideal case, represents the most similar normal feature vector to the current test feature , which is used to measure whether the current region deviates from the normal state, is the image block in the whole test image which is most different from the normal feature, that is, the position which is judged as the most possible frost layer abnormality; S543, aggregate the outliers of all image blocks, and calculate the outlier of the whole image, and the expression is: ; wherein, represents the whole image outlier result of the whole aircraft surface, which is used to quantify the overall abnormality degree of the current image, represents the image scale set participating in the aggregation, such as different resolution levels, represents a certain scale level considered at present, represents the total number of image blocks divided in the scale , represents the image content of the th image block, represents the outlier of the th image block in the scale , represents the average outlier of all image blocks in the scale , represents the weight coefficient of the scale , which is used to control the contribution proportion of different scales to the total outlier value; S544, record and store the outlier of the current detection area of the aircraft surface.

[0024] S6, acquire the image of the ground area through the color camera, and calculate the anomaly value of the ground area, the specific steps are: S61, acquire the image of the ground area through the color camera; S62, directly input the acquired ground image into the convolutional neural network model; S63, the model performs overall analysis on the input image, and outputs the anomaly value of the ground area.

[0025] S7, perform data normalization processing on the frost layer thickness calculated in step S4, the aircraft surface anomaly value calculated in step S5, and the ground area anomaly value calculated in step S6, and perform weighted fusion on the normalization processing result to calculate the final fusion value, the calculation formula is: ; In the formula, represents the fusion weight of the laser thickness measurement channel, used to control the influence degree of the frost layer thickness on the final fusion value, represents the fusion weight of the aircraft surface image anomaly value, reflecting the importance of the image channel in the final judgment, represents the fusion weight of the ground area anomaly value, used to assist in correcting the influence of environmental interference on the detection result, represents the normalized laser measured frost layer thickness value, derived from step S4, represents the normalized aircraft surface image anomaly value, derived from step S5, represents the normalized ground image anomaly value, derived from step S6; S8, judge whether the fusion value is greater than the preset threshold value, if the fusion value is less than or equal to the preset threshold value, the current detection area of the aircraft surface does not exist ice and frost, then return to execute step S4 to detect the aircraft surface in the next round; if the fusion value is greater than the preset threshold value, the current detection area of the aircraft surface exists ice and frost, the detection system will trigger the defrosting prompt alarm and perform the defrosting process.

[0026] In a preferred mode, the laser reflection acquisition in step S4, the aircraft surface image acquisition in step S5, and the ground area image acquisition in step S6 are simultaneously executed, and the same time stamp is recorded in the detection system.

[0027] Embodiment 1 S1, build an aircraft surface frost detection system, the detection system is set to a training operation mode In this embodiment, the system is built on the outdoor apron of the airport, and the surface frost layer of a parked civil aircraft is detected. The system is composed of multiple sub-modules, which are integrated on an integrated detection support. The support is installed about 1.2 meters away from the leading edge of the wing, used to fix the laser, image acquisition device and environmental sensor.

[0028] For emitting wavelength The laser beam is installed on the top of the detection bracket and tilted towards the leading edge of the wing so that the emitted laser forms an incident angle with the wing surface. In this embodiment, The laser is controlled by an independent power supply module at 532 nm, and the output power is fixed. The corresponding initial light intensity is recorded as The center height of the laser head is about 1.4 meters, ensuring that the light spot falls stably on the detection area.

[0029] The black and white industrial camera used to receive the laser reflection signal is installed below the laser. The imaging optical axis is at a non-collinear angle with the laser incident direction. The central wavelength of the camera is The narrowband filter has a bandwidth of, for example, ±10 nm, and is used to shield ambient background light. The camera has a resolution of 1280×1024 and is connected to the main control computing module.

[0030] An RGB camera for color imagery is mounted on the side of the support, with its lens facing the leading edge of the aircraft. Its field of view covers the target surface and the ground below. The camera has a resolution of 2048×2048 pixels and a fixed focal length. The captured images are used for subsequent visual anomaly detection and ground area anomaly detection.

[0031] A set of LED auxiliary light modules is installed below the camera and evenly distributed above the detection area to enhance image brightness at night or in low-light conditions. All imaging modules are connected to the embedded edge computing host via USB interfaces.

[0032] The environmental parameter acquisition module includes an integrated temperature and humidity sensor, which is installed at the same height as the lower part of the bracket and the edge of the wing, and can synchronously collect the current air temperature and relative humidity , used for subsequent environmental state estimation.

[0033] All sensor modules are connected to the edge computing unit via a unified power and data bus. The main control computing module is pre-installed with control software, which automatically launches the main detection program upon system power-up, completes data link testing, and verifies hardware status. After the system completes its power-on self-test, it has the physical foundation to enter operational mode.

[0034] S2. Offline training of the detection system is performed to collect image samples of aircraft surfaces without frost and under normal environmental conditions, and a reference library of image features of aircraft surfaces without frost is constructed. In this embodiment, the system first constructs an image feature library under the no-frost state through on-site collection before formally putting into detection. The collection site is the airport parking area, and the target object is a civil passenger plane of the same type. Under the condition of no precipitation for two consecutive days and ground temperature higher than frost point temperature, the system switches to training mode, and uses a color industrial camera to collect images of the aircraft surface. The image resolution is set to 2048x2048, the exposure time is 100 ms, 20 frames of images are collected every time period, and typical lighting environments are covered.

[0035] The system divides each image into a plurality of image blocks, each image block is denoted as , and uses the loaded convolutional neural network model to calculate the feature vector of each . The feature vector is used to represent the texture, brightness and structure distribution of the region under the no-frost state. The feature vectors of all image blocks are numbered according to their positions in the original image and organized into a standard feature set, denoted as , where each corresponds to an image block that has been confirmed as a no-frost region.

[0036] After the completion of the feature library, it is saved in binary format to the local path and is fixedly loaded for use in the system detection mode and is no longer updated. In the subsequent detection process, all image outliers are compared with this feature library as the standard.

[0037] This feature library only contains image expressions under normal working conditions and is an important reference benchmark for subsequent discrimination of image region abnormal deviation degree.

[0038] S3, the detection system is put into operation, the detection system switches the running mode to the detection mode, and the detection system is initialized In this embodiment, the system needs to complete the determination of environmental parameters and the calibration of frost layer optical properties before entering detection, which is used to support the subsequent thickness inversion model and visual judgment algorithm.

[0039] Firstly, the system collects the current environmental state through the temperature and humidity sensor installed on the detection support. On the night of January 2, 23:00, the system records the environmental temperature , and the relative humidity . All measurement data are stored in real time in the form of text in the local path.

[0040] According to the set model, the system estimates the frost point temperature of the night using the following empirical formula : ; wherein, , , , is the empirical coefficient obtained by experimental fitting in this embodiment. After substituting the data, the system calculates , and uses it as the environmental basis for subsequent frost formation determination. After completing the environmental parameter collection, the system performs calibration of the optical properties of the frost layer, including the determination of the scattering coefficient and the pore structure correction coefficient .

[0041] To measure the scattering coefficient , the operator selects a frosted plate with a known thickness D = 2.0 mm in the test area, starts the laser irradiation, and collects the reflected light intensity . After filtering and background subtraction, it is compared with the initial light intensity , and substituted into the Beer-Lambert law: ; Take the logarithmic transformation to calculate , which is used as the exponential decay term in the subsequent thickness inversion.

[0042] At the same time, to correct the non-uniform scattering caused by the existence of micro-pores inside the ice and frost, the system uses the statistical results of the previous experiments to set the pore correction coefficient . This value is stored in the configuration file of the main control program together with , and is automatically loaded in subsequent model calls.

[0043] At this point, the system has completed the environmental measurement and optical parameter calibration before detection, and is ready to enter the formal data collection process.

[0044] S4, using the principle of laser reflection and the light intensity attenuation model, calculate the frost layer thickness of the target detection area In this embodiment, the system uses oblique incidence laser irradiation and reflection image acquisition method to estimate the thickness of the frost layer on the surface of the aircraft. The wavelength of the laser is set to , the incident angle is , and the initial laser light intensity calibration value is units.

[0045] The black and white industrial camera is installed on the side of the laser, and the imaging direction forms an angle with the laser direction. A band-pass filter is fixed in front of the lens to isolate the background light. Five frames of reflection images are collected in each detection cycle. After filtering, the average brightness of the target area is extracted as the measured reflected light intensity .

[0046] According to the Beer-Lambert law, the relationship between the transmitted light intensity and the thickness is: ; where represents the transmitted light intensity after the frost layer, is the scattering coefficient, is the frost layer thickness.

[0047] The reflected light intensity caused by the frost layer can be expressed as: ; When the condition is satisfied, the exponential term can be expanded as: ; Substituting the above formula, the approximate relationship is obtained: ; Thus, the thickness can be approximately inverted from the measured reflected light intensity : ; In this embodiment, the measured reflected light intensity is , the initial intensity of the laser is , and the scattering coefficient is . Substituting the above values, we get: ; The system repeats the above inversion process 5 times and takes the average to finally obtain the frost layer thickness value , which is output as the current detection frame of the laser thickness.

[0048] S5, acquire the aircraft surface image of the current detection area by the color camera, further extract its image features, and compare with the image feature reference library of the aircraft surface in the ice-free frost state constructed in step S2, calculate the abnormal value In this embodiment, the system acquires the aircraft surface image through a color industrial camera, and evaluates the abnormality degree based on the feature comparison result of the image block and the standard sample. The image resolution is 2048x2048, and the acquisition time is synchronized with the laser irradiation.

[0049] The system divides the image into multiple image blocks, denoted as , and extracts its feature vector through a convolutional neural network. The feature vector dimension is 512, representing the texture and structure distribution of the local area. The system also calls the frost-free feature library constructed in the training stage, and calculates the following abnormal value: ; In this embodiment, the Euclidean distance between the features extracted from a certain image block and the matching sample is calculated as .

[0050] The system traverses all image blocks and performs multi-scale aggregation to calculate the full-image abnormal value ; ; Set two scales and , the outlier mean values are 0.21 and 0.31 respectively, and the corresponding weights , , the calculation is as follows: ; This value is used for subsequent fusion judgment, indicating the visual abnormality degree of the current image frame as a whole.

[0051] S6, acquire the image of the ground area by the color camera, and calculate the outlier value of the ground area In this embodiment, the system synchronously acquires the image of the ground area in each detection process, and uses it to evaluate the abnormal interference possibly introduced by the overall environmental change. The image is acquired by a fixed-angle color camera, and the resolution is set to 2048x2048. The acquisition time is consistent with the main camera.

[0052] The system inputs the ground image into a lightweight convolutional neural network model, performs end-to-end scoring, and the model outputs the ground area outlier value . In the current detection frame, the system model calculation result is: ; This outlier value does not participate in the calculation of the frost layer thickness, and is only used as an image-level auxiliary reference to improve the robustness of the system to background changes such as environmental occlusion and sudden changes in light, and is used as one of the weighted channels in the subsequent fusion decision.

[0053] S7, data normalization processing is performed on the frost layer thickness calculated in step S4, the aircraft surface outlier value calculated in step S5, and the ground area outlier value calculated in step S6, and the normalized processing result is weighted and fused to calculate the final fusion value In this embodiment, the system normalizes the laser thickness , the image outlier value and the ground area outlier value , and then weightedly combines them into a comprehensive index .

[0054] The normalization method is as follows: The values are as follows: ; ; ; The normalized result is: ; ; ; The weighting coefficient is set as , , , and the fusion value is calculated as follows: ; ; The system writes the result to the cache for subsequent judgment.

[0055] S8, judge whether the fusion value is greater than a preset threshold value, if the fusion value is less than or equal to the preset threshold value, then the current detection area of the aircraft surface does not exist frost, then return to execute step S4, the next round of detection is performed on the aircraft surface; if the fusion value is greater than the preset threshold value, then the current detection area of the aircraft surface exists frost, the detection system will trigger the defrosting prompt alarm, and defrosting The system compares the fusion value with the threshold value to determine whether there is a frosting risk.

[0056] The current fusion value is: The threshold value is set as: ; Since , the system determines that there is frost layer adhesion, and triggers the alarm module. The alarm module writes alarm records into the log, and the record content includes: 1) the current frame number; 2) , and ; 3) , and ; 4) and the judgment result; 5) the timestamp; The system also performs trend judgment on the continuous detection results. If three consecutive frames , then it is determined that there is a persistent frosting trend, and the alarm is continuously maintained; if all the consecutive frames are lower than the threshold value , then the system cancels the alarm state.

[0057] S9, closed loop control and continuous operation mechanism The system adopts a closed loop operation mode, and after initialization is completed, it enters the continuous acquisition-analysis-judgment process. Each detection cycle is 5 minutes, which is started and automatically completed by the main control program.

[0058] After each round of detection, the system stores the following data in the local record table: , and , the normalized result and , the current alarm state.

[0059] After the detection process is completed, the system immediately starts the next round of detection. The front-end interface displays the change curve and alarm state in real time, supporting all-weather operation without manual intervention.

[0060] Example 2 The following will describe the actual operation effect of the system described in the present application by combining a specific frost detection process for a parked civil aircraft. This embodiment is based on the multi-modal fusion architecture shown in Figure 2 , which tests parked aircraft at the airport. The system completes the complete process from initialization to closed-loop determination.

[0061] S1, build an aircraft surface frost detection system, set the detection system to training mode The system building architecture of this embodiment is shown in Figure 3 . The laser, black and white camera, color camera, and temperature and humidity sensor are respectively installed on the system support and aligned with the wing leading edge and ground area. Each sensor is connected to the computing host through a high-speed data cable and is configured with a synchronous trigger signal control. The system selects "detection mode" when starting and enters an automatic operation state.

[0062] S2, offline training of the detection system, collecting aircraft surface image samples under no ice frost and normal environmental conditions, and constructing an image feature reference library under no ice frost state of the aircraft surface The system collects images of the wing area under no frost conditions, extracts features from image blocks through a convolutional neural network, and constructs a standard no frost feature library. This feature library is used as a reference for subsequent detection and is fixedly stored in the database.

[0063] S3, the detection system is put into operation, the detection system switches the running mode to detection mode, and the detection system is initialized and set The laser wavelength is set to , the incident angle is set to , the initial laser intensity is unit. The color camera resolution is set to 2048x2048 pixels, the exposure time is 120 milliseconds, and the acquisition time period is from 23:00 on the current day to 06:00 on the next day. The environmental temperature is , the relative humidity is , and the frost point temperature is . According to the experimental calibration, the system sets the scattering coefficient , and the pore correction coefficient The detection area is selected as the leading edge of the wing.

[0064] S4, calculate the frost thickness of the target detection area by using the laser reflection principle and the light intensity attenuation model Laser oblique irradiation to the leading edge of the wing area, black and white camera image acquisition, after filter processing, extract the average brightness . According to the light intensity attenuation model, the frost thickness The calculation is as follows: ; The system records this value as the laser channel output .

[0065] S5, acquire the current detection area of the aircraft surface image by color camera, such as Figure 8 The RGB image collected in this embodiment is shown in the figure, further extract its image features, and compare with the image feature reference library of the aircraft surface without frost state constructed in step S2, calculate the abnormal value of the current detection area, the specific matching and memory library construction process is shown in Figure 4 .

[0066] The system synchronously acquires the aircraft surface image, which is divided into multiple image blocks . Extract the feature vector of each block, compare it with the features in the frost-free feature library, and calculate the abnormal value . After multi-scale aggregation, the full image abnormal value is obtained: ; The image presents diffuse reflection enhancement and structural distortion characteristics, and there is obvious abnormality.

[0067] S6, acquire the image of the ground area by color camera, and calculate the abnormal value of the ground area The ground area image is collected by an independent camera, and input into a lightweight convolutional neural network model to evaluate the whole image state, and the system outputs the ground area abnormal value: ; This abnormal value is input as auxiliary information of the image channel, which is used for multi-modal fusion.

[0068] S7, data normalization processing is performed on the frost thickness calculated in step S4, the aircraft surface abnormal value calculated in step S5 and the ground area abnormal value calculated in step S6, and the normalized processing result is weighted and fused to calculate the final fusion value, the fusion structure in this embodiment is shown in Figure 5 .

[0069] The system normalizes the three channel outputs respectively: ; ; ; The weighting coefficient is , , , the fusion value is calculated: ; As shown in Figure 10 , after introducing the multi-source data fusion processing mechanism, the frost thickness value obtained according to the detection result is obviously more stable, and is closer to the reference value. As shown in Figure 11 , the multi-source data fusion effectively reduces the cross entropy between the detection result and the reference value.

[0070] S8, judge whether the fusion value is greater than a preset threshold value, if the fusion value is less than or equal to the preset threshold value, there is no ice and frost on the current detection area of the aircraft surface, then return to execute step S4, and the next round of detection is performed on the aircraft surface; if the fusion value is greater than the preset threshold value, there is ice and frost on the current detection area of the aircraft surface, and the detection system will trigger a defrosting prompt alarm to defrost; The system sets a risk threshold . Since , the system determines that there is frost layer adhesion, immediately triggers an alarm signal, and records the current round of data to the log. The embodiment Figure 6 represents the frost degree change curve in the running time period, Figure 7 represents the measured frost area change curve with time.

[0071] S9, closed loop control and continuous running mechanism The system automatically performs a complete detection every 5 minutes, and continuously performs three rounds are 0.936, 0.911, and 0.922 respectively, and the sliding average is calculated: ; The system determines that the frost layer state continues to exist, maintains the alarm output and enters the next round of detection, and the closed loop runs without manual intervention. At the same time, the visual result of the frost value detected in the embodiment is as shown in Figure 9 .

[0072] The present application aims at the demand of civil aircraft surface frost detection, and constructs a multi-modal fusion scheme based on laser scattering model and visual image anomaly detection algorithm, fully considering the complex mechanism and optical properties of frost layer formation. According to the modified Beer-Lambert law, the linear relationship between frost layer thickness and diffuse reflection light intensity is established to realize real-time frost layer thickness detection. The multi-scale visual feature extraction algorithm based on convolutional neural network can quickly identify large-scale frost layer coverage and local icing anomaly. Through linear weighted fusion of the above two kinds of detection information, frost layer monitoring is realized. The system has higher robustness to environmental interference, and can trigger alarm and deicing operation in time, thereby effectively reducing the burden of aircraft maintenance personnel, and enhancing the safety and efficiency of aircraft ground operation. The system still maintains good stability under the condition of temperature fluctuation and variable light in the apron environment, and has the potential to be widely applied to the rapid detection of civil aircraft before and after parking or flight, and can provide support for the construction of smart airport.

[0073] The above-described embodiments are merely preferred embodiments of the present application, and are not intended to limit the scope of the present application. Various modifications and improvements to the technical solutions of the present application made by those of ordinary skill in the art without departing from the design spirit of the present application shall fall within the scope of protection of the present application as defined by the claims.

Claims

1. A method for detecting frost on aircraft surfaces based on laser thickness measurement and visual image analysis, characterized by: It includes the following steps: S1. Build an aircraft surface frost detection system and set the detection system to training operation mode; S2. Offline training of the detection system is performed to collect aircraft surface image samples in the frost-free state and under normal environmental conditions, and a reference library of image features of aircraft surfaces in the frost-free state is constructed; S3. The detection system is set to detection mode and the detection system is initialized; S4. Calculate the frost thickness in the target detection area using the laser reflection principle and light intensity attenuation model; S5. Capture an image of the aircraft surface in the current detection area using a color camera, extract image features, and compare the image features with the image feature reference library of the aircraft surface in a frost-free state constructed in step S2 to calculate an outlier value in the current detection area. S6. collecting an image of the ground area using a color camera and calculating an outlier value of the ground area; S7, performing data normalization processing on the frost layer thickness calculated in step S4, the aircraft surface anomaly calculated in step S5, and the ground area anomaly calculated in step S6, and performing weighted fusion on the normalized processing results to calculate the final fusion value, which is calculated as follows: ； Where, represents the fusion weight of the laser thickness measurement channel, represents the fusion weight of the outliers in the aircraft surface image, represents the fusion weight of the outliers in the ground area, represents the normalized frost thickness value measured by laser, represents the normalized aircraft surface image outlier value, represents the normalized ground image outlier value; S8. Determine whether the fusion value is greater than a preset threshold. If the fusion value is less than or equal to the preset threshold, then there is no frost in the current detection area on the aircraft surface, and the process returns to step S4 to perform the next round of detection on the aircraft surface. If the fusion value is greater than the preset threshold, then there is frost in the current detection area on the aircraft surface, and the detection system triggers a defrost warning alarm and performs defrosting.

2. The aircraft surface frost detection method based on laser thickness measurement and visual image analysis according to claim 1 is characterized in that: The laser reflection acquisition in step S4, the aircraft surface image acquisition in step S5, and the ground area image acquisition in step S6 are performed simultaneously and recorded at the same time stamp in the detection system.

3. The aircraft surface frost detection method based on laser thickness measurement and visual image analysis according to claim 1 is characterized in that: The specific steps of step S1 are: S11, arranging the laser, black and white camera, color camera, and temperature and humidity sensor on a detection system bracket, and placing the detection system bracket on one side of the aircraft wing; S12, adjusting the incident angle of the laser on the aircraft surface; S13, adjusting the field of view of the black and white camera and the color camera on the aircraft surface and the ground area; S14, connecting the laser, black and white camera, color camera, and temperature and humidity sensor to the remote server of the detection system respectively; S15. Configure a time synchronization mechanism in the detection system to synchronously complete laser measurement, image acquisition, and environmental detection parameter acquisition in each acquisition cycle; S16. Test the data integrity of each data transmission channel, and set the operation mode of the detection system to a training operation mode.

4. The method for detecting frost on an aircraft surface based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: The specific steps of step S2 are: S21. Capture images of the aircraft surface under normal environmental conditions without frost; S22, repeating step S21 at different time periods and different lighting conditions to cover a range of typical visual changes on the aircraft surface and the ground area; S23, marking all image data on the aircraft surface and the ground area collected in step S22 as having no abnormality and using them as training samples; S24, using a preset image feature extraction algorithm to encode each collected image; S25, dividing each image into multiple image blocks, and using a pre-trained convolutional neural network model to calculate feature vectors corresponding to different image blocks; S26. The feature vectors of all image blocks are stored in the image feature reference library in the order of regions, and the corresponding relationship with the original image is maintained.

5. The method for detecting frost on an aircraft surface based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: The specific steps of step S3 are: S31, setting optical parameters of the laser; S32, setting the resolution, exposure time, and brightness of the auxiliary light source of the black and white camera and the color camera; S33. Select a fixed detection area on the aircraft surface, and set the scattering coefficient, porosity correction coefficient, and frost determination threshold.

6. The method for detecting frost on an aircraft surface based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: The specific steps of step S4 are: S41, start the laser, and irradiate the target detection area with the laser according to the set incident angle; S42, after the black and white camera receives the reflected signal from the aircraft surface, it processes it through a filter to obtain an effective image signal of a single band; S43, extracting the light intensity value corresponding to the valid image in step S42, that is, the reflected light intensity actually collected by the black and white camera; S44. Calculate the transmitted light intensity of the laser after passing through the frost layer. The expression is: ； Where, represents the transmitted light intensity under ideal conditions, represents the incident light intensity, represents the exponential function, represents the frost layer scattering coefficient, Indicates the thickness of the frost layer; S45. Define the diffuse reflection amount of the frost layer to the laser, and its expression is: ； S46. Construct a linear approximation model under thin layer conditions, that is, Under these conditions, the linear approximation model is: ； Where, express Higher-order infinitesimals of S47, substituting the actual reflected light intensity collected in step S43 into the linear approximation model to calculate the frost layer thickness, which is expressed as: ； Where, Indicates the actual measured reflected light intensity; S48. Obtain multiple laser measurement values ​​and calculate the average value of the frost layer thickness.

7. The method for detecting frost on an aircraft surface based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: The specific steps of step S5 are: S51, collecting a color image of the aircraft surface in the current detection area through a color camera; S52, dividing the aircraft surface color image collected in step S51 into multiple image blocks, and using a pre-trained convolutional neural network model to calculate feature vectors corresponding to different image blocks; S53, calling the image feature reference library of the aircraft surface in the frost-free state constructed during offline training; S54: Calculate the abnormal value of the aircraft surface in the current detection area.

8. The method for detecting frost on an aircraft surface based on laser thickness measurement and visual image analysis according to claim 7, characterized in that: The specific steps of step S54 are: S541. Calculate the minimum Euclidean distance between the feature vector of each image block in the collected aircraft surface color image and the feature vector of the image in the frost-free state, and the expression is: ； Where, The feature vector representing the image block, Represents the image feature vector in the non-frost state; S542. Using a nearest neighbor feature matching and outlier calculation method, perform feature matching calculation on the collected entire aircraft surface image; S543: Aggregate the outliers of all image blocks and calculate the outlier value of the entire image. The expression is: ； Where, Indicates the outlier results of the entire aircraft surface. represents the set of image scales participating in the aggregation (e.g. different resolution levels), Indicates a scale level currently being considered, Indicated on scale The total number of image blocks divided down, Indicates the The image content of each image block, Representation scale Next, The outliers of the image blocks, Representation scale The average outlier value of all image patches under Representation scale The weight coefficient of S544. Record and store abnormal values ​​of the aircraft surface in the current detection area.

9. The method for detecting frost on an aircraft surface based on laser thickness measurement and visual image analysis according to claim 1, characterized in that: The specific steps of step S6 are: S61, collecting ground area images using a color camera; S62, directly inputting the collected ground image into the convolutional neural network model; S63. The model performs an overall analysis on the input image and outputs the outliers of the ground area.

Images