A space-time graph co-evolution network and incremental learning method for runway foreign object detection

Abstract

The application discloses a kind of pavement foreign matter detection methods of spatiotemporal graph coevolution network and incremental learning, comprising: obtaining multimodal data under pavement scene, acceleration value and angular velocity value of mobile inspection elimination and constructing carrier motion model, determine the motion state of trolley;Point cloud is compensated by rigid body transformation, multi-directional noise point diffusion defogging processing is executed to RGB image, and motion-compensated RGB-D data is generated;Space-time registration is carried out, and the multimodal image after alignment is formed;Super voxel segmentation is carried out and spatiotemporal graph is constructed, and enhanced node feature vector is generated;Input graph neural network carries out abnormal score;When detecting unknown class foreign matter, update model parameters on line;Each detection unit shares gradient by federated learning, and global collaborative detection is realized.The application significantly improves the accuracy, real-time performance and unknown target adaptability of pavement foreign matter detection, and provides strong support for ensuring the safe operation of highway, airport and other traffic.

Description

Technical Field

[0001] This invention relates to the technical field of a method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning, and particularly to a method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning. Background Technology

[0002] With the rapid expansion of transportation networks, foreign object debris (FOD) on runways has become a significant threat to driving safety. Statistics show that the economic losses caused by FOD on airport runways are enormous, and traditional manual inspection methods, due to their low efficiency, high cost, and high rate of missed detections, are no longer sufficient to meet the needs of 24 / 7 real-time monitoring.

[0003] In recent years, significant progress has been made in foreign object detection technology based on computer vision. Existing methods can be roughly divided into three categories: first, 2D image detection, which is susceptible to changes in lighting and shadows and cannot provide accurate depth information; second, 3D point cloud analysis, which can depict spatial structure but lacks temporal correlation capabilities.

[0004] More importantly, these solutions generally suffer from the following core pain points: the geometric information acquired by the depth camera is not effectively coupled with the semantic information of the RGB image, resulting in a high rate of missed detection of 2 cm × 2 cm small foreign objects against complex texture backgrounds; the field of view of a single detection unit is limited, making it difficult to form a global situational awareness, and model updates require centralized training, which cannot meet the needs of distributed real-time processing.

[0005] Therefore, in order to solve the problems of insufficient 3D-2D information fusion, poor adaptability to dynamic environment, inability to identify unknown foreign objects, lack of multi-device collaboration, and insufficient difficult sample in the existing technology, a pavement foreign object detection and incremental learning method based on 3D point cloud-spatiotemporal graph co-evolution network is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a pavement foreign object detection method based on spatiotemporal graph co-evolutionary network and incremental learning. Through spatiotemporal graph modeling and incremental meta-learning, it achieves high-precision and robust pavement foreign object detection, providing a brand-new solution for safety monitoring of intelligent transportation infrastructure.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning includes the following steps:

[0009] Step 1: Multimodal data acquisition and preprocessing:

[0010] By using a 3D depth camera mounted on a mobile inspection vehicle, RGB images, depth maps, and point cloud data of the road surface scene are collected simultaneously. The inertial measurement unit is used to obtain the carrier's motion parameters, and a motion compensation module is constructed to eliminate image blur and point cloud distortion. Spatiotemporal registration of RGB images and point clouds is performed to establish pixel-level correspondence.

[0011] Step 2: Construct a spatiotemporal graph representation:

[0012] Multi-scale geometric and semantic features of foreign objects are extracted from the preprocessed data, and a heterogeneous graph G=(V, , ), where the node set V represents the candidate target, and the spatial edge set Connecting geometrically neighboring nodes, temporal edge sets Connect adjacent frames to the same target trajectory point;

[0013] Step 3: Dynamic Adaptive Background Modeling

[0014] A multimodal background baseline based on a Gaussian mixture model is established, and the geometric distribution and RGB color distribution of the point cloud are modeled separately. IMU data is introduced to compensate for carrier motion, and the point clouds of the current frame and the reference frame are aligned using the ICP algorithm to update the background model parameters. When a sudden change in illumination or severe jitter is detected, the background update is frozen and the reference model is recorded.

[0015] Step 4: Training the Spatiotemporal Graph Co-evolutionary Network:

[0016] Constructing a graph neural network encoder-decoder structure: The encoder aggregates node neighbor information through the graph attention mechanism GAT and combines multi-head self-attention to capture long-range dependencies; the decoder reconstructs the normal appearance of nodes and predicts future motion trajectories.

[0017] Design a triple loss function: node-level reconstruction loss and the loss of combat Trajectory prediction branch loss The generator and discriminator are optimized through adversarial training;

[0018] Step 5: Incremental Meta-Learning Framework:

[0019] For unknown foreign object categories, the Model-Agnostic Meta-Learning (MAML) strategy is adopted, which freezes the basic network parameters and only updates the top-level classifier; the gradient update direction is calculated using the support set, and the effect is verified on the query set to achieve fast adaptation with a small number of samples.

[0020] Step 6: Multi-agent cooperative detection:

[0021] Multiple detection units are deployed to form a self-organizing network in the field, sharing local detection results and model gradients through wireless communication; a federated learning mechanism is used to aggregate the parameter updates of each unit and generate the globally optimal model; when a unit detects a high-confidence unknown foreign object, it broadcasts a feature descriptor to trigger synchronous learning across the entire network.

[0022] Step 7: Enhancement of Generative Hard Examples:

[0023] A conditional generative adversarial network is constructed. Normal pavement images and foreign object masks are input to synthesize difficult example samples with conditions such as motion blur and occlusion. The synthesized data is used for model fine-tuning to improve robustness in complex environments.

[0024] Step 8: Online testing:

[0025] The real-time collected data is input into the trained model to calculate the anomaly score for each node. If the score exceeds the threshold, it is determined to be a foreign object. The location, category, and size information of the foreign object are output.

[0026] Preferably, the motion compensation module in step 1 specifically includes:

[0027] A motion model of the carrier is constructed using the acceleration a(t) and angular velocity ω(t) obtained from the IMU:

[0028] ;

[0029] The rotation matrix R(t) is updated using a quaternion differential equation:

[0030] ;

[0031] in, The angular velocity read by the sensor. It is the quaternion after solving the differential equation, and R(t) is updated by the following formula:

[0032] ;

[0033] The translation vector T(t) is obtained through a second integral:

[0034] ;

[0035] in, Let be the instantaneous acceleration at time s. Zero bias of the accelerometer is obtained from static acceleration calibration. The velocity at t=0 The position of the carrier at time t=0;

[0036] Rigid body transformation compensation is performed on the point cloud using M(t), and multi-directional noise-adding diffusing convolution kernels are used for deblurring the RGB image. The loss function is multi-scale structural similarity (MS-SSIM).

[0037] .

[0038] Preferably, the spatiotemporal graph construction method in step 2 is as follows:

[0039] Perform supervoxel segmentation on the point cloud to generate candidate nodes, each node eigenvectors Geometric features With semantic features It is pieced together:

[0040] ;

[0041] Where [;] represents vector concatenation. It is the normal vector. For curvature, Relative height; This is an RGB color histogram. Texture features; temporal edges The Hungarian algorithm is used to match adjacent frame nodes, and the matching cost function is:

[0042] ;

[0043] in, For node position, and To balance the weights;

[0044] Spatial edges are constructed according to the rule that if the Euclidean distance between the centers of two nodes within the same frame is less than 0.5 m, then an undirected edge is established:

[0045] ;

[0046] in, The coordinates of the center node.

[0047] Preferably, step 3, dynamic adaptive background modeling, specifically involves:

[0048] Establish a multimodal background baseline based on a Gaussian mixture model (GMM), with the number of Gaussian components... Model the geometric distribution and RGB color distribution of the point cloud respectively:

[0049] ;

[0050] in, The number of Gaussian components. The mixing weights for the k-th Gaussian component. Let be the mean and covariance of the k-th component. Let be the probability density of a single Gaussian component. Includes node height and normal vector; model parameter update rules:

[0051] ;

[0052] in, This represents the expected change in grayscale value. This represents the magnitude of the IMU's angular velocity.

[0053] Preferably, the graph neural network encoder-decoder structure in step 4 is as follows:

[0054] The encoder uses a 4-layer graph attention network (GAT), with 8 multi-head attention heads per layer, 256 hidden dimensions, and the aggregation function is:

[0055] ;

[0056] Among them, attention coefficient The decoder is computed using an attention mechanism activated by LeakyReLU; it includes a node feature reconstruction branch and a trajectory prediction branch, with the reconstruction loss employing the L2 norm.

[0057] ;

[0058] The trajectory prediction branch uses LSTM to model the temporal dynamics, and the loss is:

[0059] ;

[0060] The adversarial loss adopts the Wasserstein GAN form, with gradient penalty coefficients. ,

[0061] ;

[0062] The weights of the total loss function are:

[0063] .

[0064] Preferably, the incremental meta-learning framework in step 5 is specifically as follows:

[0065] MAML second-order optimization was used to freeze the GAT encoder parameters. Only update the top-level classifier The inner loop update step size β = 0.01, and the outer loop learning rate η = 0.001; the support set contains K = 5 new class samples, and the query set contains 15 samples; the loss function is cross-entropy.

[0066] ;

[0067] in, The model predicts probabilities; to prevent overfitting, a weight decay λ=0.0005 is introduced.

[0068] Preferably, the multi-agent cooperative detection in step 6 includes:

[0069] Multiple detection units are deployed to form a self-organizing network in the field, sharing local detection results and model gradients via wireless communication. A federated learning mechanism is used to aggregate parameter updates from each unit, generating a globally optimal model. When a unit detects a high-confidence unknown foreign object, it broadcasts a feature descriptor to trigger synchronous learning across the entire network. The global model update weights of the federated learning aggregation mechanism are determined by the amount of data in unit i. Decide:

[0070] ;

[0071] When a unit detects a high-confidence unknown foreign object (score > 0.9), the broadcast feature descriptor is activated. Trigger network-wide synchronous incremental learning, where N is the number of detection units, and N≥3 is required.

[0072] Preferably, the conditional generative adversarial network in step 7 includes:

[0073] Generator G inputs normal track surface point cloud Foreign object mask Output synthesized point cloud :

[0074] ;

[0075] in, (0,1) represents random noise, ⊕ represents the splicing operation; the discriminator D uses Wasserstein GAN loss:

[0076] ,

[0077] Wherein, gradient penalty coefficient =10.

[0078] Preferably, step 8, online detection, includes:

[0079] The real-time collected data is input into the trained model to calculate the anomaly score for each node. If the score exceeds the threshold, it is determined to be a foreign object. The 3D location, category, and size information of the foreign object are output.

[0080] Real-time computing node anomaly scoring:

[0081] ;

[0082] Determination threshold ,when It was determined to be a foreign object at that time;

[0083] The final output information is in the format of 3D position coordinates. Category label or unknown, confidence level And the estimated size of the foreign object.

[0084] Preferably, in step 2, the multi-scale geometric features include: point cloud normal vector, curvature, and height abrupt change; the semantic features include: RGB color, texture, and context; and the node attributes include: 3D position coordinates, appearance feature vector, and motion state encoding.

[0085] The present invention achieves the following technical effects compared to the prior art:

[0086] This technology has achieved multi-dimensional performance breakthroughs in pavement foreign object detection: First, it achieves a stable recognition level for tiny foreign objects as small as 2cm×2cm, ensuring accurate capture of potential risk sources.

[0087] Secondly, the robustness of the system is significantly enhanced in the face of interference from real-world scenarios such as complex lighting conditions and motion blur, effectively overcoming the performance degradation problem of traditional detection methods in harsh environments;

[0088] Furthermore, by using a small-sample rapid adaptation mechanism for unknown foreign objects, combined with a multi-machine distributed collaborative architecture, the global detection model is dynamically updated in real time, which greatly improves the response speed and coverage of emerging foreign object types.

[0089] Ultimately, the overall technical solution fully meets the core requirements of airport scenarios for continuous 24 / 7 operation of foreign object detection, high-precision identification, and real-time early warning, providing an intelligent solution for airport pavement safety. Attached Figure Description

[0090] Figure 1 This is a diagram showing the overall architecture of the pavement foreign object detection system of the present invention.

[0091] Figure 2 This is a schematic diagram of the spatiotemporal graph construction process of the present invention;

[0092] Figure 3 This is a structural diagram of the attention network encoder of the present invention;

[0093] Figure 4 This is a schematic diagram illustrating the principle of multi-agent federated learning collaborative detection in this invention.

[0094] Figure 5 The flowchart for the conditional generation of adversarial network hard example enhancement in this invention is shown. Detailed Implementation

[0095] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0096] This invention discloses a method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning, comprising the following steps:

[0097] Step 1: Multimodal data acquisition and preprocessing:

[0098] A 3D depth camera mounted on a mobile inspection platform (such as an unmanned vehicle or drone) simultaneously acquires RGB images, depth maps, and point cloud data of the road surface scene, with a frame rate ≥30 FPS and a resolution ≥2040×1000. An inertial measurement unit (IMU, 120Hz sampling rate) is used to obtain the vehicle's acceleration a(t) and angular velocity ω(t), constructing a vehicle motion model. Rigid body transformation compensation is then performed on the point cloud.

[0099] ;

[0100] Multi-directional noise-adding diffusing convolution kernels are used to deblur RGB images, constructing a multi-scale motion blur restoration model. The loss function is minimized using multi-scale structural similarity (MS-SSIM). Spatiotemporal registration is performed between the RGB images and point clouds, and pixel-level correspondence is established using camera intrinsic and extrinsic parameter matrices to obtain aligned RGB-D data with an alignment error of <2mm.

[0101] Step 2: Construct a spatiotemporal graph representation:

[0102] The two aligned data obtained in step 1 are used to perform super-voxel segmentation (super-voxel size 5cm) on the compensated point cloud, generating a candidate node set V. For each node... Extracting multi-scale geometric features: normal vector (Calculation of local covariance matrix through principal component analysis), curvature (Through quadratic surface fitting), relative height (Relative to the pavement reference plane). Extract semantic features from the RGB image. Operations: RGB color histogram, texture feature extraction (LBP), and extraction of deep features using a pre-trained visual model (e.g., ResNet-50). Construct a heterogeneous graph G = (V, , ), where the spatial edge Connect nodes with a Euclidean distance of <0.5m, using time-series edges. The Hungarian algorithm is used to match adjacent frame nodes, with the matching cost (Cost) considering both location distance and feature similarity. Node attributes are the concatenated feature vectors, i.e., the fused feature data. .

[0103] Step 3: Dynamic Adaptive Background Modeling

[0104] Each node feature output in Step 2 is a fused high-dimensional vector, which needs to be labeled with foreground or background tags. A multimodal background baseline based on a Gaussian mixture model (GMM, 5 Gaussian components) is established, modeling the point cloud geometric distribution (normal vector, height) and RGB color distribution respectively. IMU data is introduced to compensate for carrier motion, and the point clouds of the current frame and the reference frame are aligned using the ICP algorithm, iterating until RMSE < 1cm. When a sudden change in illumination (grayscale expectation difference > 30) or severe jitter (IMU angular velocity > 5° / s) is detected, the background update is frozen and the reference model parameter α=0 is recorded. After the environment stabilizes, α=0.01 is restored and updated. The purpose is to use probabilistic modeling to screen out truly noteworthy candidate nodes from the data obtained in Step 2.

[0105] Step 4: Training the Spatiotemporal Graph Co-evolutionary Network:

[0106] The fused data features obtained from the above steps are used as the initial input to the graph neural network and are dynamically weighted and aggregated in each layer of attention calculation. The constructed graph neural network encoder-decoder structure: The encoder uses a 4-layer graph attention network (GAT), with 8 attention heads per layer, a hidden dimension of 256, and the aggregation function is:

[0107] ;

[0108] Among them, attention coefficient The attention mechanism is computed using LeakyReLU activation. Residual connections and layer normalization are introduced to stabilize training. The decoder contains two branches: a node feature reconstruction branch recovers the original features using a multilayer perceptron (MLP), with the reconstruction loss being the L2 norm. The trajectory prediction branch uses an LSTM (128 hidden layers) to model the temporal dynamics, and the loss is the Euclidean distance between the predicted and true positions.

[0109] ;

[0110] Design a weighted triple loss function:

[0111] ;

[0112] Among them, combating losses The Wasserstein GAN approach was adopted. The generator and discriminator parameters were optimized through backpropagation, with a training period of 100 epochs, a learning rate of 0.0002, and a batch size of 8.

[0113] Step 5: Incremental Meta-Learning Framework:

[0114] For unknown foreign object categories, a Model-Agnostic Meta-Learning (MAML) strategy is employed. The GAT encoder parameters are frozen, and only the top-level classifier (a 2-layer MLP) is updated. The support set contains K=5 new category samples, and the query set contains 15 samples. The inner loop update step size β=0.01, and the outer loop learning rate η=0.001. The loss function is cross-entropy, and a weight decay λ=0.0005 is introduced to prevent overfitting. Each incremental learning iteration takes less than 5 minutes, and the model parameter update volume is less than 5%.

[0115] Step 6: Multi-agent cooperative detection:

[0116] Deploy N≥3 detection units to form a self-organizing mesh network with a communication latency of <50ms. Each unit independently executes steps 1-5 and broadcasts results with local anomaly scores >0.7 to neighboring units. A federated averaging algorithm (FedAvg) is used to aggregate the model gradients of each unit, and the global model weights are updated. ,in, Let be the amount of data for unit i. When a unit detects a high-confidence unknown foreign object (score > 0.9), it triggers synchronous incremental learning across the entire network.

[0117] Step 7: Enhancement of Generative Hard Examples:

[0118] Construct a conditional generative adversarial network (cGAN), where the generator G has an encoder-decoder structure and the input is a normal track surface point cloud. Foreign object mask Output synthesized point cloud The discriminator D uses Wasserstein GAN loss and gradient penalty during training. =10, synthesizing difficult sample examples containing strong light reflection, motion blur, and partial occlusion, for model fine-tuning, thereby improving the F1-score in extreme scenarios.

[0119] Step 8: Online Detection and Early Warning:

[0120] The real-time collected data is input into the trained model to calculate the anomaly score for each node. If the value exceeds the threshold τ=0.65, it is identified as a foreign object. Output the 3D position (x, y, z accuracy ±2cm), category, and size (length × width × height) of the foreign object.

[0121] Example 1:

[0122] This embodiment discloses a method for detecting foreign objects on runways using a spatiotemporal graph co-evolutionary network and incremental learning. It is applied to the automatic inspection scenario of foreign objects (FOD) on airport runways and is mounted on a four-wheeled mobile inspection vehicle. The vehicle is equipped with a 3D depth camera (30 FPS frame rate, 2048×1024 resolution), an inertial measurement unit (IMU) (120 Hz sampling rate), and an embedded computing unit.

[0123] The specific steps are as follows:

[0124] Step 1: Multimodal data acquisition and motion compensation, spatiotemporal registration

[0125] The data acquisition and inspection vehicle travels along the airport runway at a constant speed of 8.33 m / s. The 3D depth camera simultaneously acquires RGB images, depth maps, and 3D point clouds. The IMU outputs acceleration a(t) and angular velocity ω(t) in real time for motion calculation.

[0126] The translation vector T(t) is obtained by the second integral of acceleration, and the rotation matrix R(t) is updated by the quaternion differentiation of angular velocity, thus constructing the rigid body transformation matrix M(t).

[0127] Perform rigid body transformation compensation on the point cloud to eliminate point cloud distortion caused by motion.

[0128] Multi-directional noise-adding diffusion convolution kernels are used to deblur RGB images, and MS-SSIM is used as the loss function for optimization to restore clear images.

[0129] Spatiotemporal registration utilizes camera intrinsic and extrinsic parameter matrices to perform pixel-to-point cloud registration between the compensated RGB image and the point cloud, with an alignment error of <2mm, resulting in standard RGB-D data.

[0130] Step 2: Supervoxel segmentation and spatiotemporal graph construction

[0131] Supervoxel segmentation is performed on the registered point cloud at a voxel size of 5cm to generate a candidate target node set V.

[0132] Node features are extracted as follows:

[0133] Geometric features: PCA (normal vector), curvature (quadratic surface fitting), and relative pavement height.

[0134] Semantic features: RGB color histogram, LBP texture, ResNet-50 deep features.

[0135] Geometric features and semantic feature vectors are concatenated to form the initial features of the nodes.

[0136] Spacetime graph G=(V,Es,Et)

[0137] Spatial edge Es: If the Euclidean distance between the centers of nodes within the same frame is less than 0.5m, then an undirected edge is established.

[0138] Temporal edge Et: The Hungarian algorithm is used to match adjacent frame nodes, and the cost function integrates positional distance and feature similarity to establish inter-frame trajectory association.

[0139] Step 3: Dynamic Adaptive Background Modeling

[0140] A 5-component Gaussian mixture model (GMM) is established to model the geometric distribution (height, normal vector) and RGB color distribution of the point cloud, forming a multimodal background baseline.

[0141] IMU motion compensation is introduced, and the ICP algorithm is used to align the current frame point cloud with the reference frame. The process is iterated until RMSE < 1cm, and the background parameters are updated adaptively.

[0142] When a sudden change in lighting (grayscale difference > 30) or severe jitter (angular velocity > 5° / s) is detected, background updates are frozen; updates resume after stabilization, and environmental interference is filtered out.

[0143] Step 4: Training and Anomaly Scoring of Spatiotemporal Graph Co-evolutionary Network

[0144] Encoder: 4-layer GAT graph attention network, 8-head attention, 256 hidden dimensions, aggregating neighbors and long-range dependencies.

[0145] Decoder: Two branches – node feature reconstruction branch (L2 loss) and trajectory prediction branch (LSTM, Euclidean loss).

[0146] The total loss of the triple loss function = node reconstruction loss + trajectory prediction loss + WGAN adversarial loss, weighted during training.

[0147] The anomaly scoring model outputs an anomaly score S for each node; a higher score indicates a greater probability of foreign objects.

[0148] Step 5: Incremental Meta-Learning (Rapid Adaptation to Unknown Foreign Objects)

[0149] When an unknown category of foreign object is detected (score > 0.9 and no matching label), MAML incremental learning is triggered.

[0150] Strategy: Freeze the GAT encoder and only update the top 2 layers of the MLP classifier.

[0151] Sample settings: Supports set K=5 labeled samples, query set 15 samples; inner loop step size β=0.01, outer loop learning rate η=0.001, weight decay λ=0.0005.

[0152] Stable identification of new foreign object categories can be achieved if the parameter update rate is less than 5%.

[0153] Step 6: Multi-agent federated collaborative detection

[0154] Three or more inspection units are deployed on-site, each covering 500m of the runway, forming a Mesh self-organizing network with a communication latency of <50ms.

[0155] After local detection in each unit, results with anomaly scores > 0.7 are broadcast; FedAvg federated average aggregation model gradient is used to allocate global update weights according to the amount of data.

[0156] If any unit detects a high-confidence unknown foreign object (score > 0.9), it immediately broadcasts the feature descriptor, triggering synchronous incremental learning across the entire network and achieving a unified upgrade of the global model.

[0157] Step 7: Enhanced Generative Hard Examples

[0158] Construct a conditional GAN, with the generator input being: normal pavement point cloud + foreign object mask + random noise z ~ N (0,1).

[0159] The discriminator uses WGAN-GP (gradient penalty λ=10) to synthesize difficult samples such as motion blur, 30%-70% occlusion, strong light reflection, and metallic reflection.

[0160] Fine-tuning the model with challenging case data improves the F1-score in complex scenarios and reduces the false negative rate of tiny foreign objects.

[0161] Step 8: Online detection and result output

[0162] The real-time stream is fed into the trained model to calculate node anomaly scores.

[0163] Judgment rule: If the abnormal score S>0.65, it is judged as a foreign object.

[0164] Output information:

[0165] 3D position coordinates (accuracy ±3cm), category label / unknown, confidence level,

[0166] Foreign object dimensions (length × width × height).

[0167] After completing the inspection, the system synchronizes the foreign object information to the airport runway management platform and pushes early warning information to the maintenance terminal, prompting maintenance personnel to handle the situation promptly, thus completing the entire inspection and detection closed loop. Real-world testing shows that this deployment scheme achieves a 99.2% accuracy rate for detecting pavement foreign objects with a diameter ≥3cm, with an average single-frame detection time of <100ms, meeting the real-time and accuracy requirements of online airport runway inspection.

[0168] Therefore, in scenarios where different inspection units have different detection ranges and environmental differences, the federated collaboration mechanism enables each unit to share the model's ability to detect foreign objects without sharing the original sensitive data. When a new type of foreign object appears in a certain area for the first time and completes incremental adaptation, all units can simultaneously obtain the ability to identify this type of foreign object. This avoids the problem that other nodes still miss detections after a single node has completed adaptation, effectively improving the robustness and adaptability of the overall detection in large-scale distributed inspection scenarios.

[0169] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for detecting foreign objects on pavement using a spatiotemporal graph co-evolutionary network and incremental learning, characterized in that, Includes the following steps: Step 1: Multimodal data acquisition and preprocessing: By using a 3D depth camera mounted on a mobile inspection vehicle, RGB images, depth maps, and point cloud data of the road surface scene are collected simultaneously. The inertial measurement unit is used to obtain the carrier's motion parameters, and a motion compensation module is constructed to eliminate image blur and point cloud distortion. Spatiotemporal registration of RGB images and point clouds is performed to establish pixel-level correspondence. Step 2: Construct a spatiotemporal graph representation: Multi-scale geometric and semantic features of foreign objects are extracted from the preprocessed data, and a heterogeneous graph G=(V, , ), where the node set V represents the candidate target, and the spatial edge set Connecting geometrically neighboring nodes, temporal edge sets Connect adjacent frames to the same target trajectory point; Step 3: Dynamic Adaptive Background Modeling A multimodal background baseline based on a Gaussian mixture model is established, and the geometric distribution and RGB color distribution of the point cloud are modeled separately. IMU data is introduced to compensate for carrier motion, and the point clouds of the current frame and the reference frame are aligned using the ICP algorithm to update the background model parameters. When a sudden change in illumination or severe jitter is detected, the background update is frozen and the reference model is recorded. Step 4: Training the Spatiotemporal Graph Co-evolutionary Network: Constructing a graph neural network encoder-decoder structure: The encoder aggregates node neighbor information through the graph attention mechanism GAT and combines multi-head self-attention to capture long-range dependencies; the decoder reconstructs the normal appearance of nodes and predicts future motion trajectories. Design a triple loss function: node-level reconstruction loss and the loss of combat Trajectory prediction branch loss The generator and discriminator are optimized through adversarial training; Step 5: Incremental Meta-Learning Framework: For unknown foreign object categories, the Model-Agnostic Meta-Learning (MAML) strategy is adopted, which freezes the basic network parameters and only updates the top-level classifier; the gradient update direction is calculated using the support set, and the effect is verified on the query set to achieve fast adaptation with a small number of samples. Step 6: Multi-agent cooperative detection: Multiple detection units are deployed to form a self-organizing network in the field, sharing local detection results and model gradients through wireless communication; a federated learning mechanism is used to aggregate the parameter updates of each unit and generate the globally optimal model; when a unit detects a high-confidence unknown foreign object, it broadcasts a feature descriptor to trigger synchronous learning across the entire network. Step 7: Enhancement of Generative Hard Examples: A conditional generative adversarial network is constructed. Normal pavement images and foreign object masks are input to synthesize difficult example samples with conditions such as motion blur and occlusion. The synthesized data is used for model fine-tuning to improve robustness in complex environments. Step 8: Online testing: The real-time collected data is input into the trained model to calculate the anomaly score for each node. If the score exceeds the threshold, it is determined to be a foreign object. The location, category, and size information of the foreign object are output.

2. The method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning as described in claim 1, characterized in that, The motion compensation module in step 1 specifically includes: A motion model of the carrier is constructed using the acceleration a(t) and angular velocity ω(t) obtained from the IMU: ； The rotation matrix R(t) is updated using a quaternion differential equation: ； in, The angular velocity read by the sensor. It is the quaternion after solving the differential equation, and R(t) is updated by the following formula: ； The translation vector T(t) is obtained through a second integral: ； in, Let be the instantaneous acceleration at time s. Zero bias of the accelerometer is obtained from static acceleration calibration. The velocity at t=0 The position of the carrier at time t=0; Rigid body transformation compensation is performed on the point cloud using M(t), and multi-directional noise-adding diffusing convolution kernels are used for deblurring the RGB image. The loss function is multi-scale structural similarity (MS-SSIM). 。 3. The method for detecting foreign objects on the track surface using a spatiotemporal graph co-evolutionary network and incremental learning as described in claim 1, characterized in that, The spatiotemporal graph construction method in step 2 is as follows: Perform supervoxel segmentation on the point cloud to generate candidate nodes, each node eigenvectors Geometric features With semantic features It is pieced together: ； Where [;] represents vector concatenation. It is the normal vector. For curvature, Relative height; This is an RGB color histogram. Texture features; temporal edges The Hungarian algorithm is used to match adjacent frame nodes, and the matching cost function is: ； in, For node position, and To balance the weights; Spatial edges are constructed according to the rule that if the Euclidean distance between the centers of two nodes within the same frame is less than 0.5 m, then an undirected edge is established: ； in, The coordinates of the center node.

4. The method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning as described in claim 1, characterized in that, Step 3, dynamic adaptive background modeling, specifically involves: Establish a multimodal background baseline based on a Gaussian mixture model (GMM), with the number of Gaussian components... Model the geometric distribution and RGB color distribution of the point cloud respectively: ； in, The number of Gaussian components. The mixing weights for the k-th Gaussian component. Let be the mean and covariance of the k-th component. Let be the probability density of a single Gaussian component. Includes node height and normal vector; model parameter update rules: ； in, This represents the expected change in grayscale value. This represents the magnitude of the IMU's angular velocity.

5. The method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning as described in claim 1, characterized in that, The specific structure of the graph neural network encoder-decoder in step 4 is as follows: The encoder uses a 4-layer graph attention network (GAT), with 8 multi-head attention heads per layer, 256 hidden dimensions, and the aggregation function is: ； Among them, attention coefficient The decoder is computed using an attention mechanism activated by LeakyReLU; it includes a node feature reconstruction branch and a trajectory prediction branch, with the reconstruction loss employing the L2 norm. ； The trajectory prediction branch uses LSTM to model the temporal dynamics, and the loss is: ； The adversarial loss adopts the Wasserstein GAN form, with gradient penalty coefficients. , ； The weights of the total loss function are: 。 6. The method for detecting foreign objects on the track surface using a spatiotemporal graph co-evolutionary network and incremental learning as described in claim 1, characterized in that, The incremental meta-learning framework in step 5 is specifically as follows: MAML second-order optimization was used to freeze the GAT encoder parameters. Only update the top-level classifier The inner loop update step size β = 0.01, and the outer loop learning rate η = 0.001; the support set contains K = 5 new class samples, and the query set contains 15 samples; the loss function is cross-entropy. ； in, The model predicts probabilities; to prevent overfitting, a weight decay λ=0.0005 is introduced.

7. The method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning as described in claim 1, characterized in that, Step 6, multi-agent cooperative detection, includes: Multiple detection units are deployed to form a self-organizing network in the field, sharing local detection results and model gradients via wireless communication. A federated learning mechanism is used to aggregate parameter updates from each unit, generating a globally optimal model. When a unit detects a high-confidence unknown foreign object, it broadcasts a feature descriptor to trigger synchronous learning across the entire network. The global model update weights of the federated learning aggregation mechanism are determined by the amount of data in unit i. Decide: ； When a unit detects a high-confidence unknown foreign object (score > 0.9), the broadcast feature descriptor is activated. Trigger network-wide synchronous incremental learning, where N is the number of detection units, and N≥3 is required.

8. The method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning according to claim 1, characterized in that, The conditional generative adversarial network in step 7 includes: Generator G inputs normal track surface point cloud Foreign object mask Output synthesized point cloud : ； in, (0,1) represents random noise, ⊕ represents the splicing operation; the discriminator D uses Wasserstein GAN loss: ， Wherein, gradient penalty coefficient =10.

9. The method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning as described in claim 1, characterized in that, Step 8, online detection, includes: The real-time collected data is input into the trained model to calculate the anomaly score for each node. If the score exceeds the threshold, it is determined to be a foreign object. The 3D location, category, and size information of the foreign object are output. Real-time computing node anomaly scoring: ； Determination threshold ,when It was determined to be a foreign object at that time; The final output information is in the format of 3D position coordinates. Category label or unknown, confidence level And the estimated size of the foreign object.

10. The method for detecting foreign objects on the road surface using a spatiotemporal graph co-evolutionary network and incremental learning according to claim 1, characterized in that, In step 2, the multi-scale geometric features include: point cloud normal vector, curvature, and height abrupt change; the semantic features include: RGB color, texture, and context; and the node attributes include: 3D position coordinates, appearance feature vector, and motion state encoding.

Images