Aerial track foreign matter intrusion detection method and system based on space-time feature fusion

Abstract

The application discloses a kind of based on spatiotemporal feature fusion's aerial track foreign matter invasion detection method and system.Method is through unmanned aerial vehicle collection continuous aerial video stream and pre-processing;Spatiotemporal feature fusion detection network based on improved YOLOv11 is constructed, and track foreign matter invasion detection is converted into the double target detection task of normal track area and abnormal track area.Fused-ConvGRU module is introduced in Neck module, and the spatiotemporal feature of continuous multiple frame feature maps is refined and propagated;Depth separable convolution and channel pruning strategy are used to realize model light weight.Model is trained using dataset, and real-time video stream is detected frame by frame, and alarm is generated when the confidence of abnormal track area exceeds the preset threshold value.The application does not need to preset track area and foreign matter category, can adapt to aerial view change, detection precision is high, generalization ability is strong, and light weight design is suitable for unmanned aerial vehicle embedded platform real-time deployment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent detection technology for rail transit, specifically to a method and system for detecting foreign object intrusion into rail tracks based on spatiotemporal feature fusion using aerial photography. Background Technology

[0002] With the rapid development of railway transportation, ensuring the safety of railway operations is of paramount importance. Foreign object intrusion onto the tracks (such as falling rocks, tree branches, people, and animals) is one of the main hidden dangers threatening train operation safety. Traditional inspection methods mainly rely on manual foot patrols or track vehicles, which suffer from low efficiency, high costs, and significant safety risks. In recent years, drones, with their advantages of maneuverability, wide field of vision, and low cost, have been widely used in the field of track inspection, providing a new technological means for detecting foreign object intrusion onto the tracks.

[0003] Currently, the core idea of ​​foreign object intrusion detection technology based on drone aerial video largely follows the approach used in fixed-position scenarios. This involves first manually or algorithmically defining the track area, then detecting pre-defined categories of foreign objects, and finally determining whether the foreign object is located within the track area. However, this approach has the following limitations when applied to aerial photography scenarios: First, poor applicability: In aerial videos, the number, direction, and position of tracks change in real time with the drone's flight attitude and shooting angle, making it impossible to manually pre-define fixed track areas. Second, insufficient accuracy: Existing methods mostly rely on single-frame image detection, ignoring the rich temporal information in the video stream. Foreign object intrusion is a dynamic process; utilizing spatiotemporal context information can effectively improve the detection accuracy for small-sized, moving, or partially occluded targets. Third, weak generalization ability: Foreign objects are diverse and unpredictable (e.g., construction tools, scattered goods, animals), and existing methods can only detect a few pre-defined categories, failing to handle unknown types of targets. Fourth, deployment difficulties: As an embedded platform, drones have limited computing and storage resources, making it difficult for complex deep learning models to meet the demands of real-time detection.

[0004] Therefore, there is an urgent need in this field for a lightweight orbital foreign object intrusion detection method that can adapt to the characteristics of aerial photography scenarios, does not require pre-setting of track areas and foreign object categories, can effectively utilize temporal information, and is lightweight. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for detecting foreign object intrusion on aerial photography tracks based on spatiotemporal feature fusion, so as to solve the problems of poor applicability, insufficient accuracy, weak generalization ability and difficulty in deployment on embedded platforms in the prior art.

[0006] To achieve the above objectives, the technical solution provided by this invention is: a method for detecting foreign object intrusion into aerial photography orbits based on spatiotemporal feature fusion, comprising the following steps: S1: Real-time acquisition of continuous aerial video streams containing the orbit through camera equipment mounted on the drone; preprocessing of the video stream to obtain a continuous sequence of image frames; and construction of an orbital foreign object intrusion detection dataset containing training set, validation set and test set. S2: Construct a foreign object intrusion detection network model based on an improved YOLOv11 algorithm, transforming the foreign object intrusion detection task into a dual-target detection task involving normal track regions and abnormal track regions with foreign object intrusion. The foreign object intrusion detection network model includes a Backbone network module, a Neck network module, and a Head network module connected in sequence. The Backbone network module performs multi-level convolution and downsampling on single-frame images in the image frame sequence to extract spatial semantic feature maps of different scales and output them to the Neck network module. The Neck network module receives feature maps of different scales output by the Backbone, performs cross-scale fusion of feature maps of different scales through upsampling and downsampling paths, and performs temporal information fusion of feature maps of the same scale in multiple consecutive frames through the built-in Fused-ConvGRU module to output an enhanced multi-scale spatiotemporal feature map. The Head network module performs pixel-by-pixel prediction on the enhanced multi-scale spatiotemporal feature map output by the Neck, and outputs the dual-target detection results, namely the coordinate parameters of each predicted detection box and the confidence scores of the predicted detection box belonging to the normal track region category and the abnormal track region category, respectively. S3: Train and fine-tune the foreign object intrusion detection network model using the training set and validation set to obtain a trained foreign object intrusion detection network model; test the trained foreign object intrusion detection network model using the test set to evaluate whether its performance meets the requirements. If it does not meet the requirements, continue training until the requirements are met to obtain a qualified foreign object intrusion detection network model. S4: Detect each frame of the image frame sequence using a qualified foreign object intrusion detection network model and output dual-target detection results; when the confidence of the abnormal track region in the dual-target detection results exceeds a preset threshold, it is determined that a foreign object intrusion has occurred, and an alarm message containing intrusion location information is generated based on the coordinate parameters of the predicted detection box.

[0007] To optimize the above technical solution, the specific measures also include: In step S1, the preprocessing of the video stream includes decoding, image scaling, and normalization of the video stream to adapt to the input requirements of the foreign object intrusion detection network model.

[0008] Further, in step S2, the Backbone network module includes a first DSConv layer, a second DSConv layer, a first C3k2 module, a third DSConv layer, a second C3k2 module, a fourth DSConv layer, a third C3k2 module, a fifth DSConv layer, a fourth C3k2 module, an SPPF module, and a C2PSA module connected in sequence; wherein, each DSConv layer is used for downsampling; each C3k2 module is used for feature extraction and fusion; the SPPF module is used for multi-scale pooling; and the C2PSA module is used for attention enhancement.

[0009] In step S2, the Neck network module includes a first Fused-ConvGRU module, a first Upsample layer, a first Concat layer, a fifth C3k2 module, a second Upsample layer, a second Concat layer, a sixth C3k2 module, a second Fused-ConvGRU module, a sixth DSConv layer, a seventh C3k2 module, a third Concat layer, a third Fused-ConvGRU module, a seventh DSConv layer, a fourth Concat layer, and an eighth C3k2 module connected in sequence. Each Fused-ConvGRU module is used to receive multiple consecutive frames of feature maps and perform spatiotemporal feature refinement; each Concat layer is used for feature concatenation; each C3k2 module is used for feature extraction and fusion; and each DSConv layer is used for downsampling.

[0010] Furthermore, in step S2, each Fused-ConvGRU module replaces the standard convolution with depthwise separable convolution and adds a BottleNeck module. The BottleNeck module adopts a channel bottleneck structure of first reducing dimensions and then increasing dimensions. A squeezing excitation module is added after the output of the Fused-ConvGRU module, and pointwise convolution is added between the input of the Fused-ConvGRU module and the output of the squeezing excitation module for feature fusion. The ReLU activation function is used instead of tanh as the activation function of the convolution gate.

[0011] In step S3, a multi-stage training strategy is adopted to train the foreign object intrusion detection network model. Stochastic gradient descent is used as the optimizer, and cosine annealing is used to dynamically adjust the learning rate. The training also includes channel pruning based on channel sparse training to compress the trained foreign object intrusion detection network model.

[0012] Furthermore, in step S4, the alarm information includes the intrusion time, image frame number, and coordinate parameters of the predicted detection box; the coordinate parameters are used to highlight the abnormal track area on the ground station monitoring interface.

[0013] As another important technical solution, the present invention also provides an aerial photography orbital foreign object intrusion detection system based on spatiotemporal feature fusion, comprising: The data acquisition and preprocessing module is used to acquire continuous aerial video streams containing the orbit in real time through the camera equipment mounted on the UAV, preprocess the video stream to obtain a continuous sequence of image frames, and construct an orbital foreign object intrusion detection dataset containing training set, validation set and test set. A dual-target detection network construction module is used to build a foreign object intrusion detection network model based on the improved YOLOv11 algorithm, transforming the foreign object intrusion detection task into a dual-target detection task involving normal track regions and abnormal track regions with foreign object intrusions. The foreign object intrusion detection network model includes a Backbone network module, a Neck network module, and a Head network module connected in sequence. The Backbone network module performs multi-level convolution and downsampling on single-frame images in the image frame sequence, extracting spatial semantic feature maps of different scales and outputting them to the Neck network module. The Neck network... The module receives feature maps of different scales output by the Backbone, performs cross-scale fusion of feature maps of different scales through upsampling and downsampling paths, and performs temporal information fusion of feature maps of the same scale in multiple consecutive frames through the built-in Fused-ConvGRU module, outputting an enhanced multi-scale spatiotemporal feature map; the Head network module is used to perform pixel-by-pixel prediction on the enhanced multi-scale spatiotemporal feature map output by the Neck, and outputs dual-target detection results, namely the coordinate parameters of each predicted detection box and the confidence scores of the predicted detection box belonging to the normal orbit region category and the abnormal orbit region category, respectively; The model training and performance evaluation module is used to train and fine-tune the foreign object intrusion detection network model using the training set and validation set to obtain a trained foreign object intrusion detection network model; and to test the trained foreign object intrusion detection network model using the test set to evaluate whether its performance meets the requirements. If it does not meet the requirements, training continues until the requirements are met, and a qualified foreign object intrusion detection network model is obtained. The real-time detection and alarm generation module is used to detect each frame in the image frame sequence using a qualified foreign object intrusion detection network model and output dual-target detection results. When the confidence of the abnormal track region in the dual-target detection results exceeds a preset threshold, it is determined that a foreign object intrusion has occurred, and an alarm message containing intrusion location information is generated based on the coordinate parameters of the predicted detection box.

[0014] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for detecting foreign object intrusion into aerial photography orbits based on spatiotemporal feature fusion.

[0015] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute the above-described method for detecting foreign object intrusion into aerial photography orbits based on spatiotemporal feature fusion.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention transforms the task of detecting foreign object intrusions into a dual-target detection task involving both normal and abnormal track regions. It eliminates the need for pre-delineating fixed track regions and can adapt to the dynamic changes in the number, direction, and position of tracks from the perspective of UAV aerial photography. It is applicable to diverse aerial photography scenarios, including single-track, multi-track, and complex backgrounds, significantly improving the applicability and robustness of the method.

[0017] This invention innovatively introduces the Fused-ConvGRU module into the Neck network module to refine and propagate spatiotemporal features of feature maps in multiple consecutive frames, effectively fusing temporal context information in the video stream. This significantly improves the detection capability for small, moving, or partially occluded foreign objects, reduces false alarm and false negative rates, and improves detection accuracy.

[0018] This invention only labels the state of the track area (normal or abnormal) and does not pre-define the specific category label of foreign objects. Therefore, the model does not rely on the specific types of foreign objects that appear in the training set and can effectively detect unknown types of intrusive foreign objects (such as construction tools, scattered goods, animals, etc.) that have not appeared during the training process, and has good generalization ability in open scenarios.

[0019] This invention uses depthwise separable convolutional layers (DSConv) to replace standard convolutional layers and introduces a channel pruning method based on channel sparse training to compress and accelerate the model, significantly reducing the number of model parameters and computational complexity. It can achieve real-time detection on UAV embedded platforms with limited computing and storage resources and has good engineering deployment potential. Attached Figure Description

[0020] Figure 1 This is a diagram of the track foreign object intrusion detection network structure in an embodiment of the present invention.

[0021] Figure 2 This is a structural diagram of the Fused-ConvGRU module in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the low-contribution channel pruning process in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the detection results in a single-track region scenario according to an embodiment of the present invention.

[0024] Figure 5This is a schematic diagram of the detection results in a multi-track region scenario according to an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0026] This invention provides a method for detecting foreign object intrusion into aerial orbits based on spatiotemporal feature fusion, comprising the following steps: S1: Real-time acquisition of continuous aerial video streams containing the orbit through camera equipment mounted on the drone; preprocessing of the video stream to obtain a continuous sequence of image frames; and construction of an orbital foreign object intrusion detection dataset containing training set, validation set and test set. In some implementations, a high-definition camera with a resolution of 1280×720 pixels is mounted on a DJI commercial small drone. The drone autonomously cruises along the railway line at an altitude of 4-8 meters, collecting continuous aerial video streams of the track area. The video streams are transmitted in real time to a ground station server or edge computing node via a 5G link. The ground server is equipped with an NVIDIA RTX 5880 Ada GPU for accelerating model training and inference.

[0027] Aerial videos were captured on railway tracks at different times (daytime, dusk) and in different backgrounds (urban, suburban, mountainous) using drones equipped with cameras. A dataset for detecting foreign objects intrusion on railway tracks, including tree branches, cardboard boxes, and people, was constructed. The video streams were decoded, scaled, and normalized to adapt to the input requirements of the foreign object intrusion detection network model, resulting in a continuous sequence of image frames. Each frame was labeled using an annotation tool. The annotation strategy was to label only two types of bounding boxes: normal track areas without foreign objects (labeled as "safety") and abnormal track areas with foreign objects (labeled as "danger"). No specific category label was preset for the foreign objects. The labeled dataset was then randomly divided into training, validation, and test sets according to a preset ratio (e.g., 7:2:1).

[0028] S2: Construct a foreign object intrusion detection network model based on the improved YOLOv11 algorithm, such as... Figure 1As shown, the foreign object intrusion detection task is transformed into a dual-target detection task involving both normal track regions and abnormal track regions with foreign object intrusion. The foreign object intrusion detection network model includes a Backbone network module, a Neck network module, and a Head network module connected in sequence. The Backbone network module performs multi-level convolution and downsampling on single-frame images in the image frame sequence to extract spatial semantic feature maps of different scales and output them to the Neck network module. The Neck network module receives the scale feature maps of different scales output by the Backbone, performs cross-scale fusion of feature maps of different scales through upsampling and downsampling paths, and performs temporal information fusion of feature maps of the same scale in multiple consecutive frames through the built-in Fused-ConvGRU module to output an enhanced multi-scale spatiotemporal feature map. The Head network module performs pixel-by-pixel prediction on the enhanced multi-scale spatiotemporal feature map output by the Neck, and outputs the dual-target detection results, namely the coordinate parameters of each predicted detection box and the confidence scores of the predicted detection box belonging to the normal track region category and the abnormal track region category, respectively. The Backbone network module comprises a first DSConv layer, a second DSConv layer, a first C3k2 module, a third DSConv layer, a second C3k2 module, a fourth DSConv layer, a third C3k2 module, a fifth DSConv layer, a fourth C3k2 module, an SPPF module, and a C2PSA module, connected sequentially. Each DSConv layer represents a depthwise separable convolutional layer used for downsampling; each C3k2 module is used for feature extraction and fusion; the SPPF module is used for multi-scale pooling; and the C2PSA module is used for attention enhancement.

[0029] The Neck network module comprises the following modules connected in sequence: a first Fused-ConvGRU module, a first Upsample layer, a first Concat layer, a fifth C3k2 module, a second Upsample layer, a second Concat layer, a sixth C3k2 module, a second Fused-ConvGRU module, a sixth DSConv layer, a seventh C3k2 module, a third Concat layer, a third Fused-ConvGRU module, a seventh DSConv layer, a fourth Concat layer, and an eighth C3k2 module; wherein each Concat layer is used for feature concatenation.

[0030] In some implementations, each Fused-ConvGRU module replaces standard convolution with depthwise separable convolution and incorporates a BottleNeck module. The BottleNeck module employs a channel bottleneck structure that first reduces dimensionality and then increases it. A squeezing excitation module is added after the output of the Fused-ConvGRU module, and pointwise convolution is added between the input of the Fused-ConvGRU module and the output of the squeezing excitation module for feature fusion. The ReLU activation function is used instead of tanh as the activation function for the convolution gate. The specific structure of the Fused-ConvGRU module is as follows: Figure 2 As shown, PConv represents a pointwise convolutional layer, and SE represents a squeeze excitation module. and Indicates the size of the feature map. Indicates the height of the feature map. Indicates the width of the feature map. and Both represent the number of channels in the feature map. This indicates that the number of input channels for the Fused-ConvGRU module is [number missing]. Feature map, and Indicates the first Frame image and the first Cell state of a frame image and Indicates the first Frame image and the first The hidden state of a frame image. This indicates that the number of output channels of the Fused-ConvGRU module is [number missing]. The feature map.

[0031] S3: Train and fine-tune the foreign object intrusion detection network model using the training and validation sets to obtain a trained foreign object intrusion detection network model; test the trained foreign object intrusion detection network model using the test set, and use the commonly used average precision across all classes (mAP) as the evaluation metric. When the mAP value is greater than 90%, its performance is considered to meet the requirements; if it does not meet the requirements, continue training until the requirements are met, and obtain a qualified foreign object intrusion detection network model. In some implementations, a multi-stage training strategy is employed to train the foreign object intrusion detection network model. Specifically: the first stage involves warming up the network model with the Fused-ConvGRU module based on training and validation set data; the second stage involves channel sparsity training to identify important channels; the third stage involves pruning and removing redundant channels; and the fourth stage involves fine-tuning the pruned network model. Each stage uses stochastic gradient descent as the optimizer, with an initial learning rate of 0.01 and a momentum factor of 0.9. Cosine annealing is used to decay the learning rate, training continues until the YOLOv11 loss function converges or achieves optimal performance on the validation set, and then the weight file of the foreign object intrusion detection network model is saved.

[0032] The network is trained using channel sparsity training techniques. The importance of each channel is evaluated based on the scale factor γ of the batch normalization layer, and low-contribution channels with γ values ​​close to zero are pruned and their input-output connections are removed, ultimately resulting in the network as shown below. Figure 3 The compressed network structure is shown below. The blue channels represent channels to be retained, with scaling factors γ of 1.730, 1.996, and 1.007, etc.; the green channels represent channels of the next level of depthwise separable convolutional layers; and the orange channels represent channels to be pruned, with scaling factors of 0.021 and 0.033, respectively. Indicates the first A depth-separable convolutional layer Indicates the first The first depth-separable convolutional layer One channel, Indicates the first A depth-separable convolutional layer Indicates the first The first depth-separable convolutional layer Each channel. After pruning, the model is fine-tuned, i.e., trained with a small learning rate (below 0.0001) and few epochs (generally less than 20 epochs) to restore accuracy.

[0033] S4: Detect each frame of the image frame sequence using a qualified foreign object intrusion detection network model and output dual-target detection results; when the confidence of the abnormal track region in the dual-target detection results exceeds a preset threshold, it is determined that a foreign object intrusion has occurred, and an alarm message containing intrusion location information is generated based on the coordinate parameters of the predicted detection box.

[0034] In some implementations, after the drone takes off, the camera begins to acquire a video stream in real time. The ground station server receives the video stream data, decodes the video stream into consecutive RGB frame images using a decoder, scales each frame to 640×640 pixels in real time, and performs standard preprocessing operations.

[0035] The preprocessed sequence of consecutive frames is input into a trained aerial orbital foreign object intrusion detection network model loaded with optimal weights. After passing through the Backbone, Neck, and Head network modules in sequence, the types of orbital regions in the image are obtained, and two types of detection boxes and their corresponding confidence scores are output.

[0036] In some implementations, taking the current input image size (640×640×3) as an example, the image first enters the Backbone network module for feature extraction. The image first passes through a first DSConv layer with a 3×3 kernel and a stride of 2 for feature extraction and downsampling, generating a 320×320 feature map. Continuing through a second DSConv layer, the image size becomes 1 / 4 of its original size (160×160). Then, it reaches the first C3k2 module with a 2×2 kernel and a stride of 1, where feature extraction is performed without changing the feature map size. After downsampling again through a third DSConv layer, the image size becomes 1 / 8 of its original size. Finally, after feature extraction and fusion by the second C3k2 module, an 80×80 large-scale feature map is obtained.

[0037] The 80×80 large-scale feature map is passed through the fourth DSConv layer and the third C3k2 module in sequence to obtain a 40×40 medium-scale feature map.

[0038] The 40×40 medium-scale feature map continues to pass through the fifth DSConv layer and the fourth C3k2 module, then undergoes multi-scale pooling fusion through the SPPF module and attention feature enhancement through the C2PSA module, resulting in a 20×20 small-scale feature map, thus completing the feature extraction process of the Backbone network module.

[0039] In some implementations, the Backbone network module outputs three effective feature maps at different scales: an 80×80 large-scale feature map, a 40×40 medium-scale feature map, and a 20×20 small-scale feature map. These feature maps are then fed into the Neck network module for cross-scale fusion and temporal context enhancement.

[0040] The Neck network module receives the 20×20 small-scale feature map output from the Backbone network module and inputs it into the first Fused-ConvGRU module. The first Fused-ConvGRU module receives the corresponding scale feature maps from three consecutive frames as input to the module. Through its internal depthwise separable convolutional gated recurrent units and SE attention mechanism, it refines and propagates spatiotemporal features across frames, effectively capturing the dynamic changes of intruding foreign objects, and finally outputs a 20×20 feature map processed by the first Fused-ConvGRU module. The output is upsampled through the first Upsample layer to obtain a 40×40 upsampled feature map. This upsampled feature map is then concatenated with the unsampled 40×40 mid-scale feature map in the Backbone network module along the channel dimension through the first Concat layer. Subsequently, it is fused through the fifth C3k2 module to generate a 40×40 fused feature map with stronger semantic information.

[0041] The 40×40 fused feature map is input into the second Upsample layer for upsampling to obtain an 80×80 upsampled feature map. This upsampled feature map is then concatenated with the 80×80 large-scale feature map in the Backbone network module through the second Concat layer, and then fused through the sixth C3k2 module to obtain an 80×80 high-resolution fused feature map. This feature map is rich in detailed information, which is beneficial for detecting small-sized intrusive foreign objects.

[0042] fusion of 80×80 high-resolution feature maps The input is fed into the second Fused-ConvGRU module, where the temporal context information is further refined and enhanced, resulting in an 80×80 output feature map processed by the second Fused-ConvGRU module. The processed feature map is downsampled to 40×40 pixels by the sixth DSConv layer, and then fused with the previously generated 40×40 feature map through the seventh C3k2 module. Finally, it is concatenated by the third Concat layer, and the concatenated feature map is then... The input is fed into the third Fused-ConvGRU module for temporal feature enhancement, resulting in a 40×40 output feature map after processing by the third Fused-ConvGRU module. .

[0043] The 40×40 fused feature map obtained by concatenation in the third Concat layer is downsampled to 20×20 by the seventh DSConv layer, and then concatenated with the original 20×20 small-scale feature map output by the Backbone network module (the version processed by the first Fused-ConvGRU) by the fourth Concat layer. Finally, it is fused by the eighth C3k2 module to obtain a 20×20 output feature map.

[0044] The Neck network module employs a top-down upsampling path and a bottom-up downsampling path, and introduces the Fused-ConvGRU module at key nodes for temporal feature refinement. This successfully achieves deep fusion of deep semantic information and shallow detail information across scales and time dimensions, ultimately outputting three levels of enhanced feature maps: an 80×80 feature map (corresponding to the large object detection branch of the Head network), a 40×40 feature map (corresponding to the medium object detection branch), and a 20×20 feature map (corresponding to the small object detection branch).

[0045] In some implementations, the enhanced feature maps at three scales output by the Neck network module are input into the corresponding detection branches of the Head network module. The Head network module performs bounding box regression and target classification for each grid location on each feature map using a pre-defined anchor box mechanism and a decoupled detection head. Finally, the network outputs the detection box coordinates, class labels (normal orbital regions are labeled as "safety," or abnormal orbital regions with foreign object intrusion are labeled as "danger"), and corresponding confidence scores for all predicted targets in the image.

[0046] The system presets a confidence threshold (e.g., 0.7). When the foreign object intrusion detection network model detects a frame of an image, if a detection box with the category label "danger" exists and its confidence score exceeds the preset threshold, it is determined that a foreign object intrusion event has occurred. An intrusion warning record is generated, containing information such as the intrusion occurrence time, image frame number, and the coordinates (boundary box) of the intrusion target. The "danger" area is highlighted on the ground station monitoring interface. Figure 4 and Figure 5 As shown. Ground station operators can immediately notify relevant departments for on-site verification and handling based on the alarm information.

[0047] In another embodiment of the present invention, an aerial photography track foreign object intrusion detection system based on spatiotemporal feature fusion is proposed, comprising: a data acquisition and preprocessing module, used to acquire a continuous aerial video stream containing the track in real time through a camera device mounted on a drone, preprocess the video stream to obtain a continuous image frame sequence, and construct a track foreign object intrusion detection dataset containing a training set, a validation set and a test set; A dual-target detection network construction module is used to build a foreign object intrusion detection network model based on the improved YOLOv11 algorithm, transforming the foreign object intrusion detection task into a dual-target detection task involving normal track regions and abnormal track regions with foreign object intrusions. The foreign object intrusion detection network model includes a Backbone network module, a Neck network module, and a Head network module connected in sequence. The Backbone network module performs multi-level convolution and downsampling on single-frame images in the image frame sequence, extracting spatial semantic feature maps of different scales and outputting them to the Neck network module. The Neck network... The module receives feature maps of different scales output by the Backbone, performs cross-scale fusion of feature maps of different scales through upsampling and downsampling paths, and performs temporal information fusion of feature maps of the same scale in multiple consecutive frames through the built-in Fused-ConvGRU module, outputting an enhanced multi-scale spatiotemporal feature map; the Head network module is used to perform pixel-by-pixel prediction on the enhanced multi-scale spatiotemporal feature map output by the Neck, and outputs dual-target detection results, namely the coordinate parameters of each predicted detection box and the confidence scores of the predicted detection box belonging to the normal orbit region category and the abnormal orbit region category, respectively; The model training and performance evaluation module is used to train and fine-tune the foreign object intrusion detection network model using the training set and validation set to obtain a trained foreign object intrusion detection network model; and to test the trained foreign object intrusion detection network model using the test set to evaluate whether its performance meets the requirements. If it does not meet the requirements, training continues until the requirements are met, and a qualified foreign object intrusion detection network model is obtained. The real-time detection and alarm generation module is used to detect each frame in the image frame sequence using a qualified foreign object intrusion detection network model and output dual-target detection results. When the confidence of the abnormal track region in the dual-target detection results exceeds a preset threshold, it is determined that a foreign object intrusion has occurred, and an alarm message containing intrusion location information is generated based on the coordinate parameters of the predicted detection box.

[0048] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for detecting foreign object intrusion into aerial photography orbits based on spatiotemporal feature fusion.

[0049] In another embodiment of the present invention, a computer-readable storage medium is provided, storing a computer program that causes a computer to execute a method for detecting foreign object intrusion into aerial photography orbits based on spatiotemporal feature fusion as described above.

[0050] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for detecting foreign object intrusion into aerial orbits based on spatiotemporal feature fusion, characterized in that, Includes the following steps: S1: Real-time acquisition of continuous aerial video streams containing the orbit through camera equipment mounted on the drone; preprocessing of the video stream to obtain a continuous sequence of image frames; and construction of an orbital foreign object intrusion detection dataset containing training set, validation set and test set. S2: Construct a foreign object intrusion detection network model based on the improved YOLOv11 algorithm, transforming the track foreign object intrusion detection task into a dual-target detection task of normal track regions and abnormal track regions with foreign object intrusion; the foreign object intrusion detection network model includes a Backbone network module, a Neck network module, and a Head network module connected in sequence; the Backbone network module is used to perform multi-level convolution and downsampling on single-frame images in the image frame sequence, extract spatial semantic feature maps of different scales, and output them to the Neck network module; The Neck network module receives feature maps of different scales output by the Backbone, performs cross-scale fusion of feature maps of different scales through upsampling and downsampling paths, and performs temporal information fusion of feature maps of the same scale in multiple consecutive frames through the built-in Fused-ConvGRU module, outputting an enhanced multi-scale spatiotemporal feature map; the Head network module performs pixel-by-pixel prediction on the enhanced multi-scale spatiotemporal feature map output by the Neck, and outputs dual-target detection results, namely the coordinate parameters of each predicted detection box and the confidence scores of the predicted detection box belonging to the normal orbit region category and the abnormal orbit region category, respectively; S3: Train and fine-tune the foreign object intrusion detection network model using the training set and validation set to obtain a trained foreign object intrusion detection network model; test the trained foreign object intrusion detection network model using the test set to evaluate whether its performance meets the requirements. If it does not meet the requirements, continue training until the requirements are met to obtain a qualified foreign object intrusion detection network model. S4: Detect each frame of the image frame sequence using a qualified foreign object intrusion detection network model and output dual-target detection results; when the confidence of the abnormal track region in the dual-target detection results exceeds a preset threshold, it is determined that a foreign object intrusion has occurred, and an alarm message containing intrusion location information is generated based on the coordinate parameters of the predicted detection box.

2. The method for detecting foreign object intrusion into aerial photography tracks based on spatiotemporal feature fusion according to claim 1, characterized in that: In step S2, the Backbone network module includes a first DSConv layer, a second DSConv layer, a first C3k2 module, a third DSConv layer, a second C3k2 module, a fourth DSConv layer, a third C3k2 module, a fifth DSConv layer, a fourth C3k2 module, an SPPF module, and a C2PSA module connected in sequence; wherein, each DSConv layer is used for downsampling; each C3k2 module is used for feature extraction and fusion; the SPPF module is used for multi-scale pooling; and the C2PSA module is used for attention enhancement.

3. The method for detecting foreign object intrusion into aerial photography tracks based on spatiotemporal feature fusion according to claim 1, characterized in that: In step S2, the Neck network module includes a first Fused-ConvGRU module, a first Upsample layer, a first Concat layer, a fifth C3k2 module, a second Upsample layer, a second Concat layer, a sixth C3k2 module, a second Fused-ConvGRU module, a sixth DSConv layer, a seventh C3k2 module, a third Concat layer, a third Fused-ConvGRU module, a seventh DSConv layer, a fourth Concat layer, and an eighth C3k2 module connected in sequence. Each Fused-ConvGRU module is used to receive multiple consecutive frames of feature maps and perform spatiotemporal feature refinement; each Concat layer is used for feature concatenation; each C3k2 module is used for feature extraction and fusion; and each DSConv layer is used for downsampling.

4. The method for detecting foreign object intrusion into aerial photography tracks based on spatiotemporal feature fusion according to claim 3, characterized in that: In step S2, each Fused-ConvGRU module replaces the standard convolution with depthwise separable convolution and adds a BottleNeck module. The BottleNeck module adopts a channel bottleneck structure of first reducing dimensions and then increasing dimensions. A squeezing excitation module is added after the output of the Fused-ConvGRU module, and pointwise convolution is added between the input of the Fused-ConvGRU module and the output of the squeezing excitation module for feature fusion. The ReLU activation function is used instead of tanh as the activation function of the convolution gate.

5. The aerial photography method for detecting foreign object intrusion into orbits based on spatiotemporal feature fusion according to claim 1, characterized in that: In step S3, a multi-stage training strategy is adopted to train the foreign object intrusion detection network model. Stochastic gradient descent is used as the optimizer, and cosine annealing is used to dynamically adjust the learning rate. The training also includes channel pruning based on channel sparse training to compress the trained foreign object intrusion detection network model.

6. The method for detecting foreign object intrusion into aerial photography tracks based on spatiotemporal feature fusion according to claim 1, characterized in that: In step S4, the alarm information includes the intrusion time, image frame number, and coordinate parameters of the predicted detection box; the coordinate parameters are used to highlight the abnormal track area on the ground station monitoring interface.

7. The aerial photography method for detecting foreign object intrusion into orbits based on spatiotemporal feature fusion according to claim 1, characterized in that: In step S1, the preprocessing of the video stream includes decoding, image scaling, and normalization of the video stream to adapt to the input requirements of the foreign object intrusion detection network model.

8. A system for an aerial photography orbital foreign object intrusion detection method based on spatiotemporal feature fusion as described in any one of claims 1 to 7, characterized in that, include: The data acquisition and preprocessing module is used to acquire continuous aerial video streams containing the orbit in real time through the camera equipment mounted on the UAV, preprocess the video stream to obtain a continuous sequence of image frames, and construct an orbital foreign object intrusion detection dataset containing training set, validation set and test set. A dual-target detection network construction module is used to build a foreign object intrusion detection network model based on the improved YOLOv11 algorithm, transforming the track foreign object intrusion detection task into a dual-target detection task of a normal track area and an abnormal track area with foreign object intrusion. The foreign object intrusion detection network model includes a Backbone network module, a Neck network module, and a Head network module connected in sequence. The Backbone network module is used to perform multi-level convolution and downsampling on single-frame images in the image frame sequence, extract spatial semantic feature maps of different scales, and output them to the Neck network module. The Neck network module receives feature maps of different scales output by the Backbone, performs cross-scale fusion of feature maps of different scales through upsampling and downsampling paths, and performs temporal information fusion of feature maps of the same scale in multiple consecutive frames through the built-in Fused-ConvGRU module, outputting an enhanced multi-scale spatiotemporal feature map; the Head network module performs pixel-by-pixel prediction on the enhanced multi-scale spatiotemporal feature map output by the Neck, and outputs dual-target detection results, namely the coordinate parameters of each predicted detection box and the confidence scores of the predicted detection box belonging to the normal orbit region category and the abnormal orbit region category, respectively; The model training and performance evaluation module is used to train and fine-tune the foreign object intrusion detection network model using the training set and validation set to obtain a trained foreign object intrusion detection network model; and to test the trained foreign object intrusion detection network model using the test set to evaluate whether its performance meets the requirements. If it does not meet the requirements, training continues until the requirements are met, and a qualified foreign object intrusion detection network model is obtained. The real-time detection and alarm generation module is used to detect each frame in the image frame sequence using an evaluated foreign object intrusion detection network model and output dual-target detection results. When the confidence level of the abnormal trajectory region in the dual-target detection result exceeds a preset threshold, it is determined that a foreign object intrusion has occurred, and an alarm message containing the intrusion location information is generated based on the coordinate parameters of the predicted detection box.

9. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aerial photography orbital foreign object intrusion detection method based on spatiotemporal feature fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute a method for detecting foreign object intrusion into aerial tracks based on spatiotemporal feature fusion as described in any one of claims 1 to 7.

Images