Mobile airport foreign object detection device and method based on array camera

Abstract

The invention relates to the technical field of airport safety management, in particular to a mobile airport foreign object detection device and method based on an array camera, and the device comprises an array camera module, an edge calculation module, an image detection algorithm module, a central computer module and an RTK positioning module, the edge calculation module is combined with the image detection algorithm module to process image data in real time, rapidly detect potential foreign objects and improve the detection efficiency; the RTK positioning module provides high-precision position information and provides a basis for subsequent position fusion; the central computer module fuses multi-module data; according to the technical scheme, through efficient and accurate data acquisition and processing, the detection efficiency is remarkably improved, the cost is reduced, and the phenomena of misrecognition and missing detection are effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of airport security management, and in particular to a mobile airport foreign object detection device and method based on an array camera. Background Art

[0002] Foreign objects on airport pavement, such as rocks, screws, and tools left behind during construction, pose a serious threat to flight safety and can cause tire punctures or be ingested into engines. Currently, airports primarily rely on manual inspections and fixed detection systems to prevent foreign objects. However, manual inspections are inefficient, time-sensitive, and prone to missed inspections. Fixed detection systems are expensive and only suitable for large airports, making them difficult to popularize at small and medium-sized airports. Furthermore, both manual and fixed equipment are primarily designed to address runway foreign objects. However, non-runway areas at airports are larger, harboring more foreign objects and presenting the same risk of punctures. Manual and fixed solutions struggle to balance cost, accuracy, and efficiency.

[0003] Therefore, mobile road surface foreign object detection devices are a future development trend. Existing research on mobile equipment primarily focuses on radar, which suffers from shortcomings such as misidentification of highly reflective structures (such as ground lights), poor target classification and recognition capabilities, missed detection of low-reflective targets such as rubber, and high costs. Meanwhile, image-based detection technology has rapidly developed in recent years, but in the field of road surface foreign object detection, existing methods are still primarily based on supervised learning methods. However, road surface foreign objects themselves have a limited number of samples and are difficult to enumerate, so existing methods risk missing targets that are not present in the training set.

[0004] In summary, the existing technologies for detecting foreign objects on airport pavement have the problems of low efficiency, high cost, high misidentification rate and high risk of missed detection. Summary of the Invention

[0005] The purpose of the present invention is to provide a mobile airport foreign object detection device and method based on an array camera to solve the problems of low efficiency, high cost, high misidentification rate and high risk of missed detection in the existing technology for detecting foreign objects on airport pavement.

[0006] To achieve the above-mentioned objectives, the present invention provides a mobile airport foreign object detection device based on an array camera, wherein the mobile airport foreign object detection device based on an array camera includes an array camera module, an edge computing module, an image detection algorithm module, a central computer module and an RTK positioning module. The array camera module, the edge computing module, the central computer module and the RTK positioning module are all integrated on a mobile vehicle. The array camera module is composed of 6 high-frame rate, high-resolution cameras, which are used to collect high-resolution image data of the airport road surface and transmit it to the edge computing module. The edge computing module uses the image detection algorithm module to process the image data in real time to detect potential foreign objects on the road surface. The RTK positioning module provides high-precision position information of the detection device in real time. The central computer module is responsible for receiving data from the edge computing module and the RTK positioning module, performing fusion processing, and finally outputting the precise position information of foreign objects on the road surface.

[0007] The array camera module is mounted on top of a mobile vehicle for an oblique, bird's-eye view of the runway area. Each camera in the array camera module has a resolution of 1920x1080 pixels and a maximum frame rate of 168fps. A 20cm-50cm overlap is reserved between the fields of view of every two cameras in the array camera module. The two cameras in the middle of the array camera module have the smallest focal length, while the two cameras at the edge have the largest focal length.

[0008] There are three edge computing modules, and each edge computing module corresponds to two cameras in the array camera module.

[0009] The image detection algorithm module processes the image data collected by the array camera module using an unsupervised learning method and identifies foreign objects on the road surface through an intelligent algorithm.

[0010] The present invention also provides a mobile airport foreign object detection method based on an array camera, which is applied to the mobile airport foreign object detection device based on an array camera as described above, and includes the following steps:

[0011] Data acquisition: using the array camera module to collect comprehensive image data of the airport pavement at a high frame rate and high resolution, and uploading the image data to the edge computing module;

[0012] Preliminary processing and model training: Using normal road images to synthesize a foreign object training dataset, a pre-trained ResNet network is used to train and optimize the model for binary classification. During detection, the edge computing module uses this model to perform preliminary foreign object target detection on the image data collected by the array camera module;

[0013] Positioning: Utilize the RTK positioning module to output the centimeter-level precision GPS coordinates of the device in real time;

[0014] Data fusion and target confirmation: The edge computing module is used to send preliminary foreign object target detection information to the central computer module, and the RTK positioning module is used to send the GPS coordinate information of the device to the central computer module. The central computer module confirms the world coordinate position of the foreign object target through coordinate conversion and fusion processing, and uses the target tracking algorithm for association confirmation;

[0015] Result output and subsequent processing: The central computer module outputs the world coordinate position of the foreign object and records the key detection data for subsequent analysis and optimization.

[0016] Among them, in the step of "preliminary processing and model training", the specific steps of model training include:

[0017] Dataset preparation: Normal road images are collected and divided into two parts according to a preset ratio. One part remains unchanged as background data, and the other part is used to synthesize new image data containing foreign objects;

[0018] Data synthesis: We randomly extract image patches of various sizes from public image datasets and randomly paste them onto normal road images to simulate foreign objects of different colors, textures, and sizes, thus constructing a training dataset containing a rich variety of foreign object variants.

[0019] Model selection: The pre-trained Resnet network is used as the basic model;

[0020] Binary classification training: The ResNet network is trained using a synthetic training dataset to distinguish between normal roads and roads with foreign objects. The network parameters are adjusted to optimize the model performance, enabling it to accurately identify foreign objects on the road surface.

[0021] Model evaluation and optimization: During the training process, the model is regularly evaluated using the validation set to check the accuracy and recall of the model, and the model is optimized based on the evaluation results.

[0022] Model saving and deployment: The trained model is saved as a file. When deployed, the model is loaded into the edge computing module for real-time detection of foreign objects on the road.

[0023] Among them, in the step "data fusion and target confirmation", the central computing module confirms the world coordinate position of the foreign object target through coordinate conversion and fusion processing, and uses the target tracking algorithm to perform association confirmation. The specific content is as follows: the central computing module receives target information from each camera, including the target's position (x, y) in the camera coordinate system, timestamp, and camera ID, where x and y represent the distance of the target relative to the camera. The central computing module needs to perform the following operations:

[0024] Coordinate transformation: The central computing module receives the camera target information, rotates the camera coordinate system to align it with the world coordinate system according to the heading angle of the moving vehicle, and calculates the latitude and longitude position of the target in the world coordinate system;

[0025] Target deduplication: By calculating the latitude and longitude distance between targets, targets with a distance less than 50cm are deduplicated and duplicate records are deleted;

[0026] Target association confirmation: Use the target tracking algorithm to associate and match the targets in consecutive frames to confirm whether they are the same target, and update the target status or cancel the unmatched target.

[0027] The present invention discloses a mobile airport foreign object detection device and method based on an array camera, comprising an array camera module, an edge computing module, an image detection algorithm module, a central computer module and an RTK positioning module. The array camera module is composed of six high-frame-rate, high-resolution cameras, which comprehensively capture road surface images to ensure that no objects are missed. The edge computing module, in combination with the image detection algorithm module, processes image data in real time, quickly detects potential foreign objects, and improves detection efficiency. The RTK positioning module provides high-precision position information, providing a basis for subsequent position fusion. The central computer module fuses multi-module data, accurately outputs the precise position of foreign objects, and reduces the misidentification rate and the risk of missed detection. The adoption of this technical solution significantly improves detection efficiency, reduces costs, and effectively reduces misidentification and missed detection through efficient and accurate data acquisition and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a principle block diagram of the mobile airport foreign object detection device based on array cameras provided by the present invention.

[0030] Figure 2This is a diagram showing the operating principle of the mobile airport foreign object detection device based on an array camera provided by the present invention.

[0031] Figure 3 The present invention provides a flowchart of the steps of the mobile airport foreign object detection method based on the array camera.

[0032] 101-array camera module, 102-edge computing module, 103-image detection algorithm module, 104-central computer module, 105-RTK positioning module. DETAILED DESCRIPTION

[0033] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0034] See also Figures 1 to 3 The present invention provides a mobile airport foreign object detection device based on an array camera, which includes an array camera module 101, an edge computing module 102, an image detection algorithm module 103, a central computer module 104 and an RTK positioning module 105. The array camera module 101, the edge computing module 102, the central computer module 104 and the RTK positioning module 105 are all integrated on a mobile vehicle. The array camera module 101 is composed of 6 high-frame rate, high-resolution cameras, which are used to collect high-resolution image data of the airport road surface and transmit it to the edge computing module 102. The edge computing module 102 uses the image detection algorithm module 103 to process the image data in real time to detect potential foreign objects on the road surface. The RTK positioning module 105 provides high-precision position information of the detection device in real time. The central computer module 104 is responsible for receiving data from the edge computing module 102 and the RTK positioning module 105, performing fusion processing, and finally outputting the precise position information of foreign objects on the road surface.

[0035] In this embodiment, the array camera module 101 is composed of 6 high-frame rate, high-resolution cameras, which comprehensively collect road images to ensure that there are no omissions. The edge computing module 102 is combined with the image detection algorithm module 103 to process image data in real time, quickly detect potential foreign objects, and improve detection efficiency. The RTK positioning module 105 provides high-precision position information, providing a basis for subsequent position fusion. The central computer module 104 fuses multi-module data to accurately output the precise position of foreign objects, reducing the misidentification rate and the risk of missed detection. The use of this technical solution significantly improves detection efficiency, reduces costs, and effectively reduces misidentification and missed detection through efficient and accurate data collection and processing.

[0036] Furthermore, the array camera module 101 is installed on the top of the mobile vehicle (specifically, at a height of approximately 1.5 meters from the top of the mobile vehicle) for an oblique overlooking view of the runway area. Each camera in the array camera module 101 has a pixel resolution of 1920x1080 and a maximum frame rate of 168fps. A 20cm-50cm overlap is reserved between the fields of view of every two cameras in the array camera module 101. In the array camera module 101, the two cameras located in the middle have the smallest focal lengths, while the two cameras located at the edges have the largest focal lengths.

[0037] In this embodiment, the array camera module 101 configuration and layout are adopted, and the high resolution (1920x1080 pixels) and high frame rate (up to 168fps) imaging capabilities of each camera are used to ensure clear and smooth images. The 20cm-50cm field of view overlap between cameras is combined to optimize coverage and resource utilization. The differentiated design of the middle camera with a small focal length provides a wide field of view for rapid positioning, while the edge cameras with a large focal length focus on a local area to achieve high-precision recognition effectively takes into account the breadth and precision of foreign object detection on airport roads, significantly improving the comprehensiveness and accuracy of the detection system.

[0038] Furthermore, the number of the edge computing modules 102 is 3, and each edge computing module 102 corresponds to two cameras in the array camera module 101.

[0039] In this embodiment, by setting up three edge computing modules 102 and making each edge computing module 102 correspond to two cameras in the array camera module 101, distributed parallel processing of image data collected by high-resolution array cameras is achieved. This modular and parallel design effectively improves data processing efficiency, reduces the load pressure of a single module, and at the same time enhances the scalability and fault tolerance of the system, ensuring that the airport road foreign object detection system can operate in real time, efficiently and stably.

[0040] Furthermore, the image detection algorithm module 103 uses an unsupervised learning method to process the image data collected by the array camera module 101 and recognize foreign objects on the road surface through an intelligent algorithm.

[0041] In this embodiment, the image data collected by the array camera module 101 is processed by the image detection algorithm module 103 using an unsupervised learning method, thereby achieving the effect of automatically identifying foreign objects on the road using an intelligent algorithm without manual labeling. This automated and intelligent processing method not only improves detection efficiency, but also reduces dependence on human intervention, and enhances the system's adaptability and detection accuracy in complex environments.

[0042] See also Figure 3 The present invention also provides a mobile airport foreign object detection method based on an array camera, which is applied to the mobile airport foreign object detection device based on an array camera as described above, and includes the following steps:

[0043] Data acquisition: using the array camera module 101 to collect comprehensive image data of the airport pavement at a high frame rate and high resolution, and uploading the image data to the edge computing module 102;

[0044] Preliminary processing and model training: A foreign object training dataset is synthesized using normal road images, and a pre-trained ResNet network is used to train and optimize the model for binary classification. During detection, the edge computing module 102 uses this model to perform preliminary foreign object target detection on the image data collected by the array camera module 101;

[0045] Position positioning: Utilize the RTK positioning module 105 to output the GPS coordinates of the device in real time with centimeter-level accuracy;

[0046] Data fusion and target confirmation: The edge computing module 102 is used to send preliminary foreign object target detection information to the central computer module 104, and the RTK positioning module 105 is used to send the GPS coordinate information of the device to the central computer module 104. The central computer module 104 confirms the world coordinate position of the foreign object target through coordinate conversion and fusion processing, and uses the target tracking algorithm to perform association confirmation;

[0047] Result output and subsequent processing: The central computer module 104 outputs the world coordinate position of the foreign object and records key detection data for subsequent analysis and optimization.

[0048] In this embodiment, .

[0049] Furthermore, in the step of "preliminary processing and model training", the specific steps of model training include:

[0050] Dataset preparation: Normal road images are collected and divided into two parts according to a preset ratio. One part remains unchanged as background data, and the other part is used to synthesize new image data containing foreign objects;

[0051] Data synthesis: We randomly extract image patches of various sizes from public image datasets and randomly paste them onto normal road images to simulate foreign objects of different colors, textures, and sizes, thus constructing a training dataset containing a rich variety of foreign object variants.

[0052] Model selection: The pre-trained Resnet network is used as the basic model;

[0053] Binary classification training: The ResNet network is trained using a synthetic training dataset to distinguish between normal roads and roads with foreign objects. The network parameters are adjusted to optimize the model performance, enabling it to accurately identify foreign objects on the road surface.

[0054] Model evaluation and optimization: During the training process, the model is regularly evaluated using the validation set to check the accuracy and recall of the model, and the model is optimized based on the evaluation results.

[0055] Model saving and deployment: The trained model is saved as a file. When deployed, the model is loaded into the edge computing module 102 for real-time detection of foreign objects on the road surface.

[0056] In this implementation, given the difficulty in obtaining airport FOD images, normal pavement images are directly used to synthesize a foreign object (FOD) training dataset. During dataset creation, the original dataset without FOD is split into two parts at a 5:5 ratio. One part remains unchanged, while the other part is used to synthesize new images containing FOD.

[0057] The data synthesis process is as follows: first, randomly extract image blocks of different sizes from various public image datasets; second, randomly paste the image blocks onto the road surface image to simulate foreign objects of arbitrary color, texture, and size.

[0058] During model training, the pre-trained Resnet network was used, and the above-made dataset was used for binary classification training to obtain the optimized neural network model.

[0059] Preferably, the specific steps in the step "binary classification training" include the following:

[0060] (1) Initialize the neural network model and target tracker;

[0061] (2) Preprocess the input image, including image resizing and brightness adjustment;

[0062] (3) The image is input into the trained neural network model to extract the feature map and output the binary classification result of the image;

[0063] (4) For images that are judged to contain foreign objects in the binary classification, the feature map is connected to Grad-CAM or Lightweight CAM to generate a heat map. The heat map can represent the importance score of each pixel position. The pixels with more obvious colors are most likely to be foreign objects:

[0064] CAM=Σ(αk*Ak);

[0065] Among them, αk is the weight of the k-th channel, Ak is the k-th feature map;

[0066] (5) Use adaptive threshold (Otsu algorithm) to segment the heat map, cluster the heat map areas, and then perform binarization on the heat map to generate a black and white image, where the darker areas in the heat map are represented as white and the others are represented as black.

[0067] T=argmax(σ 2 (t));

[0068] Among them, T is the optimal threshold, σ 2 (t) is the between-class variance, and the white area means that there is a high possibility of foreign objects;

[0069] (6) Perform connected domain analysis on the black and white image to filter out areas with too small an area;

[0070] (7) Calculate the IOU between the target in the white area and the target in the previous 10 frames of video to form an IOU matrix. Use the Hungarian algorithm to match to confirm whether the current target and the target in the previous frame belong to the same target. The matching algorithm uses the IOU matrix as the cost matrix to solve the optimal match:

[0071] minΣ(Cij*Xij);

[0072] Where Cij is the cost matrix (-IOU), cost = 1-IOU, Xij is the matching matrix;

[0073] (8) Finally, the matching results are processed. If IOU>0.3, the match is considered successful, the target position is updated, and the disappearance counter is reset. If the target is not matched for 10 consecutive frames, it is automatically deregistered.

[0074] Furthermore, in the step "data fusion and target confirmation", the central computing module confirms the world coordinate position of the foreign object target through coordinate conversion and fusion processing, and uses the target tracking algorithm to perform association confirmation. The specific content is as follows: the central computing module receives target information from each camera, including the target's position (x, y) in the camera coordinate system, timestamp, and camera ID, where x and y represent the distance of the target relative to the camera. The central computing module needs to perform the following operations:

[0075] Coordinate transformation: The central computing module receives the camera target information, rotates the camera coordinate system to align it with the world coordinate system according to the heading angle of the moving vehicle, and calculates the latitude and longitude position of the target in the world coordinate system;

[0076] Target deduplication: By calculating the latitude and longitude distance between targets, targets with a distance less than 50cm are deduplicated and duplicate records are deleted;

[0077] Target association confirmation: Use the target tracking algorithm to associate and match the targets in consecutive frames to confirm whether they are the same target, and update the target status or cancel the unmatched target.

[0078] In this embodiment, since the array camera module 101 includes six cameras, there is some overlap between the cameras. It is possible for two cameras to simultaneously report the same target. The target reported by each camera includes the target's position (x, y) in the camera's coordinate system, a timestamp, and a camera ID. x and y represent the target's distance from the camera.

[0079] First rotate the camera coordinate system to align with the world coordinate system:

[0080]

[0081] Where θ represents the heading angle of the vehicle.

[0082] Next, calculate the x' of each camera target i and y' i The corresponding latitude and longitude arc:

[0083]

[0084] Among them, i represents the i-th foreign object target, Δlon represents the accuracy difference value, Δlat represents the dimension difference value, R represents the radius of the earth, lat i Represents the current vehicle's latitude.

[0085] Finally, the latitude and longitude of each target in the world coordinate system can be expressed as:

[0086] lon' i =Δlon+loni ;

[0087] lat' i =Δlat+lat i ;

[0088] Finally, calculate the latitude and longitude distance between each target to remove duplicate targets:

[0089] D i,j =F([lon′ i ,lat′ i ],[lon′ j ,lat′ j ]);

[0090] If the distance value is less than 50cm, it is considered to be the same target. After deleting the duplicate targets, the latitude and longitude of all foreign objects are output.

[0091] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A mobile airport foreign object detection device based on an array camera, characterized in that: It includes an array camera module, an edge computing module, an image detection algorithm module, a central computer module and an RTK positioning module. The array camera module, the edge computing module, the central computer module and the RTK positioning module are all integrated on a mobile vehicle. The array camera module consists of 6 high-frame rate, high-resolution cameras, which are used to collect high-resolution image data of the airport road surface and transmit it to the edge computing module. The edge computing module uses the image detection algorithm module to process the image data in real time to detect potential foreign objects on the road surface. The RTK positioning module provides high-precision position information of the detection device in real time. The central computer module is responsible for receiving data from the edge computing module and the RTK positioning module, performing fusion processing, and finally outputting the precise position information of foreign objects on the road surface.

2. The mobile airport foreign object detection device based on an array camera according to claim 1, characterized in that: The array camera module is mounted on top of a mobile vehicle and is used to provide an oblique downward view of the runway area. Each camera in the array camera module has a resolution of 1920x1080 pixels and a maximum frame rate of 168fps. A 20-50cm overlap is reserved between the fields of view of every two cameras in the array camera module. The two cameras in the middle of the array camera module have the smallest focal length, while the two cameras at the edge have the largest focal length.

3. The mobile airport foreign object detection device based on an array camera according to claim 2, characterized in that: The number of the edge computing modules is 3, and each edge computing module corresponds to 2 cameras in the array camera module.

4. The mobile airport foreign object detection device and method based on array camera according to claim 3, characterized in that: The image detection algorithm module uses an unsupervised learning method to process the image data collected by the array camera module and identifies foreign objects on the road through an intelligent algorithm.

5. A mobile airport foreign object detection method based on an array camera, applied to the mobile airport foreign object detection device based on an array camera as claimed in claim 1, characterized in that: The steps include: Data acquisition: using the array camera module to collect comprehensive image data of the airport pavement at a high frame rate and high resolution, and uploading the image data to the edge computing module; Preliminary processing and model training: Using normal road images to synthesize a foreign object training dataset, a pre-trained ResNet network is used to train and optimize the model for binary classification. During detection, the edge computing module uses this model to perform preliminary foreign object target detection on the image data collected by the array camera module; Positioning: Utilize the RTK positioning module to output the centimeter-level precision GPS coordinates of the device in real time; Data fusion and target confirmation: The edge computing module is used to send preliminary foreign object target detection information to the central computer module, and the RTK positioning module is used to send the GPS coordinate information of the device to the central computer module. The central computer module confirms the world coordinate position of the foreign object target through coordinate conversion and fusion processing, and uses the target tracking algorithm for association confirmation; Result output and subsequent processing: The central computer module outputs the world coordinate position of the foreign object and records the key detection data for subsequent analysis and optimization.

6. The mobile airport foreign object detection device and method based on array camera according to claim 3, characterized in that: In the "Preliminary Processing and Model Training" step, the specific steps of model training include: Dataset preparation: Normal road images are collected and divided into two parts according to a preset ratio. One part remains unchanged as background data, and the other part is used to synthesize new image data containing foreign objects; Data synthesis: We randomly extract image patches of various sizes from public image datasets and randomly paste them onto normal road images to simulate foreign objects of different colors, textures, and sizes, thus constructing a training dataset containing a rich variety of foreign object variants. Model selection: The pre-trained Resnet network is used as the basic model; Binary classification training: The ResNet network is trained using a synthetic training dataset to distinguish between normal roads and roads with foreign objects. The network parameters are adjusted to optimize the model performance, enabling it to accurately identify foreign objects on the road surface. Model evaluation and optimization: During the training process, the model is regularly evaluated using the validation set to check the accuracy and recall of the model, and the model is optimized based on the evaluation results. Model saving and deployment: The trained model is saved as a file. When deployed, the model is loaded into the edge computing module for real-time detection of foreign objects on the road.

7. The mobile airport foreign object detection device and method based on array camera according to claim 3, characterized in that: In the step "Data Fusion and Target Confirmation," the central computing module confirms the world coordinate position of the foreign object target through coordinate conversion and fusion processing, and uses the target tracking algorithm to perform association confirmation. Specifically, the central computing module receives target information from each camera, including the target's position (x, y) in the camera coordinate system, a timestamp, and a camera ID, where x and y represent the distance of the target relative to the camera. The central computing module needs to perform the following operations: Coordinate transformation: The central computing module receives the camera target information, rotates the camera coordinate system to align it with the world coordinate system according to the heading angle of the moving vehicle, and calculates the latitude and longitude position of the target in the world coordinate system; Target deduplication: By calculating the latitude and longitude distance between targets, targets with a distance less than 50cm are deduplicated and duplicate records are deleted; Target association confirmation: Use the target tracking algorithm to associate and match the targets in consecutive frames to confirm whether they are the same target, and update the target status or cancel the unmatched target.

Images