Airport apron fixed FOD identification method based on image identification technology

Abstract

The invention provides an airport apron fixed FOD recognition method based on an image recognition technology, and relates to the technical field of FOD recognition, and the method comprises the steps: synchronously collecting a visible light image, infrared thermal imaging and laser radar point cloud data of an airport apron region; performing illumination equalization and geometric correction preprocessing on the acquired data; a fixed FOD and a temporary moving object are separated through dynamic background modeling, and the modeling fuses visible light, infrared characteristics and point cloud space distribution; a deep learning model is adopted to fuse the multispectral features and perform classification and positioning, and a channel attention mechanism is embedded in the model; processing data through an edge computing device, triggering grading alarm in real time according to a classification result, and recording a log; and dynamically updating the deep learning model parameters. Through a three-in-one framework of multi-modal data fusion, self-adaptive dynamic modeling and edge intelligent calculation, four core problems of high omission ratio, high false alarm rate, poor environmental adaptability and insufficient real-time performance in existing FOD detection are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of FOD identification, and in particular to a method for identifying apron-fixed FOD based on image recognition technology. Background Art

[0002] Foreign Object Debris (FOD) is one of the major hidden dangers threatening aviation safety, especially small objects such as metal parts and plastic fragments fixed to the apron that are easily overlooked.

[0003] Current mainstream detection methods generally include manual inspections and single image sensors. Manual inspections are not only inefficient and have high false positive rates, but also lack coverage in complex weather conditions (such as rain, fog, and at night). Single sensors are susceptible to interference from lighting and shadows, making it difficult to reliably identify small or low-contrast FOD. Furthermore, conventional deep learning models adapted for existing single image sensors (such as YOLO and Faster R-CNN) lack generalization capabilities in dynamic and complex scenes.

[0004] In addition, existing related technologies lack a multimodal data fusion mechanism and cannot effectively distinguish between fixed FOD and temporary moving objects (such as vehicles and personnel). Summary of the Invention

[0005] In order to solve at least some of the technical problems in the related art, the present invention provides a method for identifying apron-fixed FOD based on image recognition technology.

[0006] In order to achieve the above object, the technical solution adopted by the present invention includes: The present invention provides a method for identifying apron-fixed FOD based on image recognition technology, comprising the following steps: Step S1: synchronously collect visible light images, infrared thermal images and lidar point cloud data of the apron area; Step S2: pre-processing the collected data by illumination equalization and geometric correction; Step S3: Separating fixed FOD and temporary moving objects through dynamic background modeling, wherein the modeling integrates visible light, infrared features and point cloud spatial distribution; Step S4: using a deep learning model to fuse multispectral features and perform classification and positioning, wherein the model is embedded with a channel attention mechanism; Step S5: Process the data through the edge computing device, trigger a hierarchical alarm in real time based on the classification results, and record the log; Step S6: Dynamically update deep learning model parameters.

[0007] Optionally, in step S3, the dynamic background modeling adopts a joint probability model: Where, Indicates that given a feature vector Background under conditions The probability density function of represents the number of Gaussian distributions in the Gaussian mixture model and , Indicates the Gaussian distribution weight coefficients, and satisfy , Indicates is the mean, is the Gaussian distribution probability density function of the covariance matrix, For the The mean of a Gaussian distribution, For the The covariance matrix of a Gaussian distribution, Represents the pixel feature vector, which is composed of visible light image features and infrared thermal imaging features Composition, that is , Represents the weight coefficient of the uniform distribution term of the point cloud space, represents the uniform distribution of point cloud space and , is the volume of the space where the point cloud is located.

[0008] Optionally, in step S3, the temporary moving object is excluded if: Where, represents the length of the observation time window, Indicates time The collected LiDAR point cloud coordinate set, Indicates time The collected LiDAR point cloud coordinate set, Indicates the time interval, is the displacement threshold.

[0009] Optionally, in step S4, the calculation of the channel attention mechanism includes: Where, represents the channel weight vector generated by the channel attention mechanism, represents the Sigmoid activation function, represents the weight matrix of the second fully connected layer, represents the ReLU activation function, represents the weight matrix of the first fully connected layer, represents the global average pooling operation, represents the input feature map, which is the input of the channel attention module, represents the feature map after recalibration by the channel attention mechanism, Indicates the channel index of the feature map.

[0010] Optionally, the multispectral features are fused by the following formula: Where, represents the fused feature map, Represents a 1×1 convolution operation, used for feature fusion and dimensionality reduction, represents the feature map extracted from the visible light image, represents the feature map extracted from the infrared thermal imaging image, Represents the feature map extracted from depth data (converted from LiDAR point cloud).

[0011] Optionally, the total delay of processing data by the edge computing device satisfies: Where, represents the total system latency on the edge computing device, represents the time consumption of the image preprocessing stage, Indicates the number of floating-point operations required for model inference, Indicates the computing power of the edge device GPU, Indicates the time consumption of the Non-Maximum Suppression operation.

[0012] Optionally, the model parameters are dynamically updated via online learning: When the false positive rate is greater than 5% for 30 consecutive frames, an update is triggered; Where, represents the updated model parameters, represents the current model parameters, represents the learning rate and , represents the gradient of the loss function with respect to the model parameters, represents the loss function, Represents a newly collected dataset for online learning.

[0013] Beneficial effects: 1. Through the above technical solution, the present invention solves the four core problems of existing FOD detection, namely, high missed detection rate, high false alarm rate, poor environmental adaptability, and insufficient real-time performance, through the "three-in-one" architecture of multimodal data fusion + adaptive dynamic modeling + edge intelligent computing.

[0014] Specifically, first, step S1 of the present invention synchronizes visible light images, infrared thermal imaging, and lidar point cloud data to build a multispectral, spatial, and thermal radiation joint perception system. Step S2 uses illumination equalization and geometric correction preprocessing to eliminate light / shadow interference and improve data consistency. This not only effectively achieves all-weather coverage but also effectively enhances anti-interference capabilities.

[0015] Second, step S3 of the present invention fuses the spatial distribution of visible light, infrared light, and point cloud data through dynamic background modeling to construct a spatiotemporal joint probability model. This not only accurately separates fixed and moving targets, but also effectively reduces the false alarm rate.

[0016] Third, the present invention can not only effectively enhance the small target detection capability, but also enhance the generalization capability of dynamic scenes through the channel attention mechanism + multispectral feature fusion in step S4 and the dynamic update model parameters in step S6.

[0017] Fourth, step S5 of the present invention can achieve not only millisecond-level response but also effective resource optimization through edge computing device processing + hierarchical alarm.

[0018] 2. Other beneficial effects or advantages of the present invention will be described in detail in the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] in: Figure 1 This is a schematic flow chart of steps of a method for identifying fixed FOD on an apron based on image recognition technology provided by an exemplary embodiment of the present invention; Figure 2 1 is a schematic diagram of a system for implementing the method for identifying fixed FOD on an apron based on image recognition technology according to an exemplary embodiment of the present invention; Figure 3is a schematic diagram of a multimodal image fusion process provided by an exemplary embodiment of the present invention; Figure 4 is a schematic diagram of dynamic background modeling and FOD separation provided by an exemplary embodiment of the present invention; Figure 5 It is a structural diagram of a deep learning model provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0022] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0023] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0024] like Figures 1 to 5 As shown, according to a first aspect of the present invention, a method for identifying apron fixed FOD based on image recognition technology is provided, comprising the following steps: Step S1: synchronously collect visible light images, infrared thermal images and lidar point cloud data of the apron area; Step S2: pre-processing the collected data by illumination equalization and geometric correction; Step S3: Separate fixed FOD and temporary moving objects through dynamic background modeling, and model and fuse visible light, infrared features and point cloud spatial distribution; Step S4: Use a deep learning model to fuse multispectral features and perform classification and positioning, and embed the model into a channel attention mechanism; Step S5: Process the data through the edge computing device, trigger a hierarchical alarm in real time based on the classification results, and record the log; Step S6: Dynamically update deep learning model parameters.

[0025] Through the above technical solution, the present invention solves the four core problems of existing FOD detection, namely "high missed detection rate, high false alarm rate, poor environmental adaptability, and insufficient real-time performance", through the "three-in-one architecture of multimodal data fusion + adaptive dynamic modeling + edge intelligent computing".

[0026] Specifically, first, step S1 of the present invention synchronizes visible light images, infrared thermal imaging, and lidar point cloud data to construct a multi-spectral-spatial-thermal radiation joint perception system. Step S2 performs illumination equalization and geometric correction preprocessing to eliminate illumination / shadow interference and improve data consistency. In this way, not only can all-weather coverage be effectively achieved (infrared penetrates rain, fog, and smoke, lidar ignores illumination changes, and visible light retains details. The three complement each other to completely resolve detection blind spots in complex weather conditions such as nighttime and rainy fog), but also an effective leap in anti-interference capability can be achieved (through multi-source data correction, the impact of illumination mutations, shadows, reflections, etc. on the recognition of small / low-contrast FODs can be significantly reduced).

[0027] Second, step S3 of the present invention fuses the spatial distribution of visible light, infrared light, and point cloud data through dynamic background modeling to construct a spatiotemporal joint probability model. This not only accurately separates fixed and moving targets, but also effectively reduces the false alarm rate.

[0028] Third, the present invention can not only effectively enhance the small target detection capability through the channel attention mechanism + multispectral feature fusion in step S4 and the dynamic update of model parameters in step S6 (channel attention can enhance the thermal radiation characteristics in the infrared channel and the texture characteristics in the visible light channel, thereby improving the detection rate of tiny FOD such as screws and gravel), but also enhance the generalization capability of dynamic scenes (online learning enables the model to adapt to unknown scenes such as airport pavement wear, seasonal changes, new FOD, etc., avoiding the static model failure problem of traditional YOLO / Faster R-CNN).

[0029] Fourth, step S5 of the present invention, through edge computing device processing + hierarchical alarming, can not only achieve millisecond-level response (through edge GPU computing power and lightweight preprocessing, the time from data collection to alarm triggering is less than 100ms to meet the real-time standards of aviation safety), but also effectively achieve resource optimization (hierarchical alarming to significantly reduce the operation and maintenance load, for example, metal parts → emergency alarm, leaves → log recording).

[0030] In one embodiment of the present invention, in step S3 of the present invention, the dynamic background modeling adopts a joint probability model: Where, Indicates that given a feature vector Background under conditions The probability density function of represents the number of Gaussian distributions in the Gaussian mixture model and , Indicates the Gaussian distribution weight coefficients, and satisfy , Indicates is the mean, is the Gaussian distribution probability density function of the covariance matrix, For the The mean of a Gaussian distribution, For the The covariance matrix of a Gaussian distribution, Represents the pixel feature vector, which is composed of visible light image features and infrared thermal imaging features Composition, that is , Represents the weight coefficient of the uniform distribution term of the point cloud space, represents the uniform distribution of point cloud space and , is the volume of the space where the point cloud is located.

[0031] In this embodiment, the calculation formula In terms of is the pixel feature vector, which is the visible light image feature and infrared thermal imaging features The joint feature vector of is used to fuse texture + thermal radiation features; is the number of Gaussian distributions ( ), which are used to model background, foreground (FOD), and shadow / noise respectively; The uniform distribution term of the point cloud space is used to characterize the invariance of the object's spatial position (the fixed FOD position remains unchanged); is the point cloud constraint weight coefficient (usually 0.3 to 0.5), which is used to balance the contribution of spectral features and spatial position.

[0032] In this embodiment, the method of the present invention can not only reduce the false alarm rate to the extreme and achieve sub-pixel level small target detection and positioning, but also effectively maintain robustness in dynamic scenes. In addition, it can also ensure computational efficiency and meet real-time requirements.

[0033] The technical principle of this embodiment is described below.

[0034] 1. How does this invention solve the problem of light / shadow interference (a defect of traditional GMM)? The traditional GMM (Gaussian mixture model) only relies on RGB or infrared single features, and is prone to misjudging shadows as foreground objects under strong shadows. Dual-spectrum joint modeling, where visible light Capturing texture, infrared Perceived temperature: shadows appear dark in visible light, but their temperature is consistent with the surroundings. This is represented by the covariance matrix in the joint distribution. Negative correlation, so that the probability of the shadow area can be Significantly higher than the actual FOD, thereby reducing the shadow misjudgment rate.

[0035] 2. How does the present invention exclude temporary moving objects (i.e., moving objects such as vehicles and people that have similar spectral characteristics to FOD in a single-frame image): This invention introduces point cloud space constraints , so that for a fixed FOD, its position remains unchanged, and the corresponding The spatial distribution is consistent in consecutive frames, that is, The value is stable; for moving objects, their positions will change, and the corresponding There will be drastic fluctuations in consecutive frames, resulting in the overall Based on this, we can combine the judgment logic (such as the exclusion conditions of temporary moving objects as described below) ), the point cloud determined to be a moving object is excluded, thereby effectively reducing the false alarms caused by moving objects.

[0036] 3. How does the present invention enhance the detection rate of small-sized FOD? Its essence is to solve the problem that small FOD (such as bolts) are easily submerged by noise in traditional background modeling due to their low pixel ratio. The value of , to set up a Gaussian distribution to model small targets (weight Adaptive adjustment), so that the lidar is sensitive to small protrusions (for example, those with a height greater than 0.5 cm will be detected), The term maps the physical size constraint to the image domain to improve the detection rate of small-sized FOD.

[0037] The above implementation is described below in conjunction with a specific application scenario.

[0038] Scenario: Detecting metal screws on the apron after rain.

[0039] The difficulty of the scene lies in the fact that the reflection of accumulated water interferes with visible light, and the low temperature rain reduces the infrared contrast.

[0040] The present invention's approach: First, for the visible light channel For the infrared channel, the reflection is suppressed by balancing the illumination. As for the point cloud, since the metal screw absorbs heat quickly, there is a certain temperature difference between it and the cold water; in addition, For example, the screw protrusion height is about 0.8cm. The trigger probability of the item is increased.

[0041] Result: Screw , determined as foreground FOD, and the surrounding water area , determined to be background.

[0042] In one embodiment of the present invention, in step S3 of the present invention, the temporary moving object is excluded if: Where, represents the length of the observation time window, Indicates time The collected LiDAR point cloud coordinate set, Indicates time The collected LiDAR point cloud coordinate set, Indicates the time interval, is the displacement threshold.

[0043] In this embodiment, the calculation formula In terms of For in time The collected LiDAR point cloud coordinate set is used to represent the absolute position of the object in physical space; The length of the observation time window (usually 30 to 60 seconds) is used to cover the passage cycle of typical moving objects (vehicles / people) at the airport; It is the time interval for point cloud collection, which is used to balance real-time performance and displacement accuracy; is the Euclidean distance, which is used to calculate the maximum spatial displacement between point cloud sets (not point displacement or average displacement); is the displacement threshold, which is a rigid standard for determining the mobility of an object.

[0044] In this embodiment, the method of the present invention can effectively solve the vehicle / personnel interference problem through the maximum displacement criterion + absolute space measurement, and at the same time, through the point cloud displacement stability ( ), which can ensure that the real FOD is not incorrectly filtered. In addition, the lidar can penetrate rain, fog, and smoke, ensuring that the judgment criteria can still be reliably executed under complex weather conditions. In addition, the algorithm complexity is only O(N), and it can complete the cloud computing of 10,000 points within 5ms on the edge GPU, which can effectively ensure real-time performance.

[0045] The technical principle of this embodiment is described below.

[0046] 1. How does the present invention solve the interference caused by moving objects (e.g., vehicles, people, etc.): In existing technologies, image-based optical flow methods or inter-frame difference methods can easily misjudge slow-moving objects (such as luggage carts) as fixed FOD. However, this invention can effectively solve the problem of false positives for moving objects. Specifically, taking vehicles and people as an example, for moving vehicles, the contact points between their tires and the ground are significantly located, which will cause the overall tire point cloud displacement to be much greater than the displacement threshold. , so that it can be accurately marked to achieve vehicle exclusion; for moving people, their footsteps cause the displacement of the foot point cloud to be much greater than the displacement threshold , so that it can be filtered in real time to exclude people.

[0047] 2. How does the present invention prevent fixed FOD from being mistakenly removed? Strong winds can cause FOD such as leaves or dropped rubber gaskets to vibrate. Conventional methods may misjudge them as moving objects due to pixel fluctuations. However, in the method of the present invention, the overall displacement of the leaves or rubber gaskets (not a single point) is detected. , based on the maximum displacement criterion of the present invention (the single point displacement may be large, but the maximum displacement of the overall point cloud is still lower than the threshold), it is possible to prevent fixed FOD from being mistakenly cleared.

[0048] 3. How to ensure the robustness of detection for small-sized FOD: In existing technologies, when a vehicle passes near a small FOD (e.g., a nut), traditional methods will lose the FOD due to image occlusion. However, in this invention, after identifying a moving object, it is marked (see the vehicle marking process in the "How to Solve Moving Objects" section above) and its occluded area is dynamically masked. This ensures that the displacement of the occluded FOD point cloud within the time window is always zero, maintaining its fixed properties, thereby ensuring robust detection of small FOD.

[0049] For a comparison between this embodiment and the optical flow method in the prior art, please refer to the following Table 1 Table 1 Comparison between the present invention and optical flow Comparison Project Optical flow method Point cloud displacement criterion of the present invention Slow-moving luggage cart Mistakenly identified as stationary (speed < threshold) Correct exclusion (wheel displacement > τ) Metal gaskets in strong winds Mistakenly identified as motion (pixel jitter) Correct preservation (point cloud displacement < τ) Screws under vehicle cover Missed detection (occlusion causes feature disappearance) Continuous tracking (point cloud position remains unchanged) Ground reflection in rainy days Lots of false positives (optical flow noise) Unaffected (laser penetrates rain and fog) Computational complexity High (dense matching required) Low (point cloud distance calculation can be accelerated in parallel) In one embodiment of the present invention, in step S4 of the present invention, the calculation of the channel attention mechanism includes: Where, represents the channel weight vector generated by the channel attention mechanism, represents the Sigmoid activation function, represents the weight matrix of the second fully connected layer, represents the ReLU activation function, represents the weight matrix of the first fully connected layer, represents the global average pooling operation, represents the input feature map, which is the input of the channel attention module, represents the feature map after recalibration by the channel attention mechanism, Indicates the channel index of the feature map.

[0050] In this embodiment, the calculation formula In terms of The input feature map is the original expression of multimodal features (including visible light / infrared / depth features); It is a global average pooling operation, which is used to compress spatial information and generate channel descriptors; is the weight of the first fully connected layer, which is used to reduce the dimension and extract the nonlinear relationship between channels; It is the ReLU activation function, which introduces nonlinearity and suppresses invalid channels; is the weight of the second fully connected layer, which is used to restore the channel dimension and learn the channel importance; Is the Sigmoid activation function, which normalizes the weight to , identifies the importance of the channel; is the channel weight vector, which quantifies the contribution of each channel to FOD detection; It is a recalibrated feature map, which is used to enhance key channel features and suppress noise channels.

[0051] In this embodiment, the method of the present invention can not only effectively improve the detection rate of micro FOD, but also resolve multimodal conflicts and achieve adaptive environment switching.

[0052] Specifically, in order to improve the detection rate of micro FOD, in the existing related technology, bolts with a diameter of less than 2 cm are detected in the visible light channel. It only occupies 2×2 pixels. At the same time, it has Since metal has low emissivity and is easily drowned by the ambient thermal noise, the existing technology has a low detection rate for tiny FOD. In the present invention, in, is the visible light channel weight vector, is the infrared channel weight vector, is the depth channel weight vector.

[0053] Take two exemplary application scenarios as examples: Scenario 1: Nighttime metal gasket detection: The weight is automatically increased (large temperature difference in the environment), Automatically reduce (insufficient light), so that the infrared characteristics can be enhanced and the visible light noise can be suppressed, thereby achieving accurate detection of metal gaskets at night.

[0054] Scenario 2: Detection of transparent plastic sheets in rainy weather: The weight is automatically increased (the point cloud height is abnormal), Automatically reduce (reflective interference) and use depth features to dominate recognition, thereby achieving accurate detection of transparent plastic sheets in rainy days.

[0055] Regarding the effect of resolving multimodal conflicts, in existing related technologies, the reflection of accumulated water appears as a bright area in the visible light channel (similar to metal FOD) and a low temperature in the infrared channel (the opposite of real metal FOD). In this way, there is a multimodal feature conflict. The present invention can resolve the multimodal feature conflict through the attention mechanism. For example, first By calculating the channel mean, we can learn that for the water area, the visible light channel mean is high and the infrared channel mean is low, while for the real metal, the visible light channel mean and the infrared channel mean are both high at the same time. 、 ) can learn this conflict pattern and output synchronously lower visible light channel mean and infrared channel mean. In this way, the conflicting features are suppressed and false alarms can be effectively avoided.

[0056] In one embodiment of the present invention, the multispectral features of the present invention are fused by the following formula: Where, represents the fused feature map, Represents a 1×1 convolution operation, used for feature fusion and dimensionality reduction, represents the feature map extracted from the visible light image, represents the feature map extracted from the infrared thermal imaging image, Represents the feature map extracted from depth data (converted from LiDAR point cloud).

[0057] In this embodiment, the present invention uses a 1×1 convolution operation to fuse the channel splicing results of visible light features, infrared features and depth features, thereby realizing cross-modal feature interaction (automatically learning weight distribution through convolution kernels), so that metal parts are mainly identified by infrared features in rainy and foggy days, and transparent plastic fragments are enhanced by depth features to locate them, thereby improving the detection rate of tiny FOD; and, by only performing linear combination of channel dimensions (without spatial convolution), the amount of calculation is reduced to 1 / 9 of the traditional 3×3 convolution, and the edge fusion delay is compressed to <5ms (meeting the real-time standard of aviation safety); at the same time, by retaining the original resolution of multiple modalities and aligning pixels, the conflicting modal weights are automatically suppressed in interference scenarios such as accumulated water reflection, thereby reducing the false alarm rate; in addition, the absolute spatial support is provided by the depth feature, and even if the optical sensor fails (such as blinding due to strong light), the system availability can still be maintained through point cloud height anomalies.

[0058] In one embodiment of the present invention, the total delay of processing data by the edge computing device of the present invention satisfies: Where, represents the total system latency on the edge computing device, represents the time consumption of the image preprocessing stage, Indicates the number of floating-point operations required for model inference, Indicates the computing power of the edge device GPU, Indicates the time consumption of the Non-Maximum Suppression operation.

[0059] In this embodiment, by setting the total delay formula in the edge computing device and constraining the preprocessing delay and NMS operation delay, not only can the total system delay be compressed to less than 25ms, but also the end-to-end response triggered by the FOD alarm can meet the relevant standards, thereby reducing the possibility of major accidents caused by FOD.

[0060] In one embodiment of the present invention, the model parameters of the present invention are dynamically updated through online learning: When the false positive rate is greater than 5% for 30 consecutive frames, an update is triggered; Where, represents the updated model parameters, represents the current model parameters, represents the learning rate and , represents the gradient of the loss function with respect to the model parameters, represents the loss function, Represents a newly collected dataset for online learning.

[0061] In this implementation, by adopting an online learning update strategy and setting the false positive rate > 5% continuous frame as the trigger condition, not only can the model automatically adapt to the seasonal changes of the airport pavement (such as reflection of water in the rainy season and shrinkage of asphalt in winter), but also the false alarm rate can be kept at a low level during continuous operation. In addition, the learning rate This can ensure the stable convergence characteristics of the model; thereby reducing the frequency of manual parameter adjustment and lowering the difficulty and cost of operation and maintenance.

[0062] The above are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. The method for identifying fixed FOD on the apron based on image recognition technology is characterized by: The steps include: Step S1: synchronously collect visible light images, infrared thermal images and lidar point cloud data of the apron area; Step S2: pre-processing the collected data by illumination equalization and geometric correction; Step S3: Separating fixed FOD and temporary moving objects through dynamic background modeling, wherein the modeling integrates visible light, infrared features and point cloud spatial distribution; Step S4: using a deep learning model to fuse multispectral features and perform classification and positioning, wherein the model is embedded with a channel attention mechanism; Step S5: Process the data through the edge computing device, trigger a hierarchical alarm in real time based on the classification results, and record the log; Step S6: Dynamically update deep learning model parameters.

2. The method for identifying apron-fixed FOD based on image recognition technology according to claim 1, characterized in that: In step S3, the dynamic background modeling adopts a joint probability model: Where, Indicates that given a feature vector Background under conditions The probability density function of represents the number of Gaussian distributions in the Gaussian mixture model and , Indicates the Gaussian distribution weight coefficients, and satisfy , Indicates is the mean, is the Gaussian distribution probability density function of the covariance matrix, For the The mean of a Gaussian distribution, For the The covariance matrix of a Gaussian distribution, Represents the pixel feature vector, which is composed of visible light image features and infrared thermal imaging features Composition, that is , Represents the weight coefficient of the uniform distribution term of the point cloud space, represents the uniform distribution of point cloud space and , is the volume of the space where the point cloud is located.

3. The method for identifying apron-fixed FOD based on image recognition technology according to claim 2, characterized in that: In step S3, the temporary moving object is excluded if: Where, represents the length of the observation time window, Indicates time The collected laser radar point cloud coordinate set, Indicates time The collected laser radar point cloud coordinate set, Indicates the time interval, is the displacement threshold.

4. The method for identifying apron-fixed FOD based on image recognition technology according to claim 1, characterized in that: In step S4, the calculation of the channel attention mechanism includes: Where, represents the channel weight vector generated by the channel attention mechanism, represents the Sigmoid activation function, represents the weight matrix of the second fully connected layer, represents the ReLU activation function, represents the weight matrix of the first fully connected layer, represents the global average pooling operation, represents the input feature map, which is the input of the channel attention module, represents the feature map after recalibration by the channel attention mechanism, Indicates the channel index of the feature map.

5. The method for identifying apron-fixed FOD based on image recognition technology according to claim 1, characterized in that: The multispectral features are fused by the following formula: Where, represents the fused feature map, Represents a 1×1 convolution operation, used for feature fusion and dimensionality reduction, represents the feature map extracted from the visible light image, represents the feature map extracted from the infrared thermal imaging image, Represents the feature map extracted from depth data (converted from LiDAR point cloud).

6. The method for identifying apron-fixed FOD based on image recognition technology according to claim 1, characterized in that: The total latency of processing data by edge computing devices satisfies: Where, represents the total system latency on the edge computing device, represents the time consumption of the image preprocessing stage, Indicates the number of floating-point operations required for model inference, Indicates the computing power of the edge device GPU, Indicates the time consumption of the Non-Maximum Suppression operation.

7. The method for identifying apron-fixed FOD based on image recognition technology according to claim 1, characterized in that: The model parameters are dynamically updated through online learning: When the false positive rate is greater than 5% for 30 consecutive frames, an update is triggered; Where, represents the updated model parameters, represents the current model parameters, represents the learning rate and , represents the gradient of the loss function with respect to the model parameters, represents the loss function, Represents a newly collected dataset for online learning.

Images