Railway foreign matter video generation and monitoring method based on text driving and feature collaboration

Abstract

The invention discloses a railway foreign matter video generation and monitoring method based on text driving and feature collaboration. The method comprises the following steps: 1) acquiring low-resolution and high-resolution features through a backbone network; using the low-resolution feature to drive the high-resolution feature to perform background reconstruction and calculating the difference with the current frame feature to obtain a spatial difference; 2) driving the high-resolution feature to perform time reconstruction through the low-resolution feature, recovering a spatial difference, and calculating a difference with a current frame spatial difference to obtain a time difference; 3) obtaining a foreign matter probability graph through operations such as bilinear interpolation, and then carrying out binarization processing to obtain a foreign matter binary graph; 4) updating the background image and calculating corresponding features, and replacing the longest feature in the feature queue to form a new queue; and 5) calculating the loss between the foreign matter probability graph and the true value by using the cross entropy, performing verification on the verification set, performing statistics on the common failure scene, generating the railway foreign matter video in a targeted manner, amplifying the existing data, and improving the adaptability of the algorithm. According to the method, the F1 score of video monitoring reaches 0.909 in the railway scene under the condition that the annotation is reduced by 80%.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent video surveillance technology, specifically relating to a method for generating and monitoring railway foreign object videos based on text-driven and feature-coordinated approaches. Background Technology

[0002] Foreign object intrusion detection is one of the core technologies for ensuring train operation safety. Existing methods mainly rely on traditional image processing algorithms or deep learning models, but they face the following limitations in complex scenarios: 1. High dependence on labeled data and extremely time-consuming: Current mainstream methods (such as object detection models based on convolutional neural networks) rely on large-scale labeled video data for supervised training. However, the diversity of railway scenes (such as different weather and lighting conditions) leads to high labeling costs, and manual labeling is difficult to cover all potential abnormal events (such as rockfalls and pedestrian intrusions). According to statistics, labeling a single 1,000-frame video segment takes more than 120 hours, which seriously restricts the model's generalization ability.

[0003] 2. Limited accuracy, particularly insufficient generalization to noise interference: Railway scenes contain a large amount of dynamic background interference (such as swaying vegetation, passing trains, and sudden changes in light and shadow), making it difficult for traditional methods (such as background difference methods and optical flow methods) to effectively separate foreground objects from the dynamic background. For example, the false detection rate of inter-frame difference-based methods is as high as 12.4% in swaying vegetation scenes, while optical flow-based methods are sensitive to changes in illumination.

[0004] Video foreign object detectors suffer from the problem of difficulty in transferring across datasets and are highly dependent on labeled data: existing methods rely on large-scale manually labeled video data, which is costly and difficult to cover complex scenes (such as rainy or snowy weather, or insufficient lighting at night); traditional algorithms (such as background subtraction) have the limitation of being unable to separate foreground foreign objects (such as falling rocks) from dynamic background interference (such as swaying vegetation or passing trains), resulting in a high false detection rate. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide a method for generating and monitoring foreign objects in railways based on text-driven and feature-based collaborative technologies. This method integrates low-resolution driven reconstruction, dual-group attention mechanism, temporal consistency video generation, and dynamic noise suppression technology to solve problems such as high false detection rate, difficulty in generating long-term data, and insufficient real-time performance in railway scenarios. It is suitable for the intelligent upgrading and efficient deployment of railway safety monitoring systems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: Step 1: Spatial attention mechanism for low-resolution driving high-resolution reconstruction: Input a sequence of railway monitoring video frames, and focus on the current frame image... Low-resolution features are obtained through the backbone network. Features with high resolution ; Use low-resolution features to drive high-resolution features for background reconstruction and calculate the spatial difference with the features of the current frame. ,in, It is the number of channels. It is the width of the current frame image. It is the height of the current frame image. It is the field of real numbers; Specifically, high-resolution features are extracted through the Pixel_Shuffle operation. Converted to the same spatial size as low resolution To optimize computational efficiency, all reconstruction formulas are performed using low-resolution features, as designed below: Low-resolution features of the current frame In the The value at each position corresponds to a eigenvectors; Low-resolution features of the current frame In the The value at each position corresponds to a eigenvectors; Features In the The value at each position corresponds to a eigenvectors; and These are transformation matrices for the query variable and the key variable, both of which are The transformation matrix; It is a value transformation matrix, which is The transformation matrix is ​​then used to form the attention matrix, as follows: in, Features The transpose of the given form is then normalized using the Softmax operation: in, It is an exponential function, whose goal is to recover background information from other location information, as follows: This recovered background feature Then, through the Pixel_Shuffle operation, it is transformed into high-resolution reconstructed background features: in The reconstructed background features have the following spatial differences from the current frame features: .

[0007] Step 2, Temporal Attention Mechanism for Low-Resolution-Driven High-Resolution Reconstruction: Based on the stored historical feature queue of railway surveillance videos, the low-resolution features obtained in Step 1 are used to drive the high-resolution features and perform temporal reconstruction, recovering spatial differences from the temporal axis, and calculating the spatial difference with the features of the current frame. The difference in time of acquisition ; Specifically, based on the stored historical feature queue of railway surveillance videos, the low-resolution features obtained in step 1 are used to drive the high-resolution features for temporal reconstruction, and the spatial difference features with the current frame are calculated. The differences were used to obtain time difference characteristics. Spatial differences in step 1 Converted to the same spatial size as low resolution To optimize computational efficiency, the time axis reconstruction formulas are all performed under low-resolution features, as designed below: in, A queue of historical features can be used to depict the dynamic changes in the background. For storage Low-resolution features at any given moment; This represents the high-resolution features at that moment; This represents the spatial difference characteristics at that moment; This involves transforming spatial difference features into low-resolution features using the Pixel_Shuffle operation. Furthermore, Low-resolution features of the current frame In the The value at each position corresponds to a The eigenvectors; similarly For low-resolution features in the historical queue in the first The value at each position corresponds to a eigenvectors, Spatial difference features in historical queues In the The value at each position corresponds to a eigenvectors; and These are transformation matrices for the query variable and the key variable, both of which are The transformation matrix; It is a value transformation matrix, which is The transformation matrix is ​​then used to form the attention matrix, as follows: in, yes The transpose of the given form is then normalized using the Softmax operation: Indicates position i This characteristic is consistent with the historical queue. The probability value of generating similarity values ​​for features at the same location at the same time, where, It is an exponential function; the background information recovered from historical information is as follows: This recovered spatial difference feature Then, through the Pixel_Shuffle operation, it is transformed into spatial difference features for high-resolution reconstruction, as follows: The temporal difference between the reconstructed spatial difference features and the spatial difference features of the current frame is as follows: in, This is due to time differences.

[0008] Step 3: Calculate and optimize the probability map of foreign object intrusion limit: address the time difference generated in Step 2. The image is enlarged to its original size using bilinear interpolation, and then... The convolution combined with the Sigmoid function yields the foreign object probability map. Then, binarization is performed, followed by morphological post-processing with 3×3 erosion and 5×5 dilation operations to remove isolated noise and restore the integrity of the foreign object's outline, resulting in the final foreign object segmentation result, which is a binary image of the same size as the original image. ; Step 4, Dynamic Queue Update and Noise Suppression: Update the dynamic background queue. A new background image is calculated using the current frame and the foreign object segmentation result. For this new background image, the backbone network calculates its corresponding features and replaces the oldest feature in the feature queue, resulting in a new feature queue. Specifically, for each new input frame, a new background is generated based on the monitoring results, as follows: in, It is a smoothing factor, usually chosen as 0.95. The background image stored from the previous moment is an image of the same size as the input image frame. It's a pixel-by-pixel multiplication operation. The binary image corresponding to the size of the original image is the foreign object monitoring result generated in step 4.

[0009] To address this new context, a backbone network is used to compute its corresponding features, and the oldest feature in the feature queue is replaced to obtain a new feature queue, represented as: in, Stored in the history queue Background image of the time stamp. In the historical queue Low-resolution features of time ticks High-resolution features at the same time point, This represents the spatial difference characteristics at that moment. Features are spatial difference features after undergoing the Pixel_Unshuffle operation.

[0010] Step 5, Network Training and Text-Based Video Augmentation: For the generated foreign object probability map, cross-entropy loss is used to calculate the foreign object probability map and the ground truth value. The losses between them are as follows: in, It is cross-entropy loss.

[0011] Simultaneously, validation is performed on the validation set, statistically analyzing failure scenarios under common weather conditions, such as rain, snow, and changes in lighting. Based on the target text descriptions constructed from the failure scenarios and the original video sequences, videos of foreign objects on railways are generated using the ControlNet module to augment the existing dataset and serve as enhanced samples for subsequent training, thereby improving the samples' adaptability to working conditions.

[0012] This invention proposes a method for generating and monitoring railway foreign object intrusion videos based on text-driven and dynamic feature collaboration. Compared with existing technologies, the beneficial effects are as follows: Optimization of computational efficiency: By employing a spatial attention mechanism and a temporal attention mechanism (steps 1 and 2) that drive high-resolution reconstruction from low resolution, the computational load for feature extraction and background reconstruction is reduced while maintaining high fidelity. Traditional methods have a processing latency of 80ms / frame on surveillance videos, while this invention reduces the latency to ≤35ms / frame, meeting the requirements of real-time railway monitoring.

[0013] Improved detection accuracy: The time attention mechanism (step 2) driven by low resolution and reconstructed by high resolution is used in conjunction with dynamic queue update and noise suppression (step 4) to effectively separate foreground foreign objects from dynamic background interference.

[0014] Meanwhile, the temporal attention mechanism (step 2) that drives high-resolution reconstruction from low resolution, combined with network training and text-based video enhancement (step 5) techniques, significantly improves the generalization ability to noise interference such as vegetation swaying, passing trains, and sudden changes in light and shadow. Experiments show that on a test set containing 30% dynamic interference, the monitoring accuracy improved from 0.852 to 0.923.

[0015] Improved long-video monitoring capabilities: Through dynamic queue updates and noise suppression (step 4) and network training and text-based video enhancement (step 5), the current model cannot adapt to long-term monitoring scenarios that can cover all-weather railway monitoring scenarios.

[0016] Reduced annotation requirements: Network training and text-based video enhancement (step 5) automatically generate diverse foreign object intrusion scene data (such as rain, snow, and nighttime lighting), reducing the need for manual annotation by 80% compared to traditional methods. Attached Figure Description

[0017] Figure 1 This is a flowchart for calculating the probability of foreign object intrusion.

[0018] Figure 2 This is a schematic diagram of low-resolution feature extraction driving high-resolution structure.

[0019] Figure 3 It is a dynamic queue maintenance and update graph.

[0020] Figure 4 It is a flowchart of text-driven video generation.

[0021] Figure 5 This is a screenshot of the video's generated effect. Detailed Implementation

[0022] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0023] Figure 1 This is a flowchart for calculating the probability of foreign object intrusion. Specifically, to achieve the detection of foreign objects in the video, it can be divided into three parts: (1) Spatial attention mechanism of low resolution driving high resolution reconstruction (2) Temporal attention mechanism for low-resolution driving high-resolution reconstruction (3) Calculate and optimize the foreign object intrusion probability map.

[0024] in: The spatial attention mechanism that drives high-resolution reconstruction from low resolution is as follows: Figure 2 As shown, by combining low-resolution and high-resolution features, computational complexity is reduced while computational accuracy is improved. For the current frame image Low-resolution features are obtained through the backbone network. Features with high resolution ; Use low-resolution features to drive high-resolution features for background reconstruction and calculate the spatial difference with the features of the current frame. ,in, It is the number of channels. It is the width of the current frame image. It is the height of the current frame image. It is the field of real numbers.

[0025] Specifically, high-resolution features are extracted through the Pixel_Shuffle operation. Converted to the same spatial size as low resolution To optimize computational efficiency, all reconstruction formulas are performed using low-resolution features, as designed below: Low-resolution features of the current frame In the The value at each position corresponds to a eigenvectors; Low-resolution features of the current frame In the The value at each position corresponds to a eigenvectors; Features In the The value at each position corresponds to a eigenvectors; and These are transformation matrices for the query variable and the key variable, both of which are The transformation matrix; It is a value transformation matrix, which is The transformation matrix is ​​then used to form the attention matrix, as follows: in, Features The transpose of the given form is then normalized using the Softmax operation: in, It is an exponential function, whose goal is to recover background information from other location information, as follows: This recovered background feature Then, through the Pixel_Shuffle operation, it is transformed into high-resolution reconstructed background features: in The reconstructed background features have the following spatial differences from the current frame features: .

[0026] The temporal attention mechanism for low-resolution driving high-resolution reconstruction adopts the following approach: Figure 2 A similar process. Based on the stored historical feature queue of railway surveillance videos, the low-resolution features obtained in step 1 are used to drive high-resolution features and reconstruct them along the time axis. Spatial differences are recovered from the time axis, and the spatial differences with the features of the current frame are calculated. The difference in time of acquisition .

[0027] Specifically, based on the stored historical feature queue of railway surveillance videos, the low-resolution features obtained in step 1 are used to drive the high-resolution features for temporal reconstruction, and the spatial difference features with the current frame are calculated. The differences were used to obtain time difference characteristics. Spatial differences in step 1 Converted to the same spatial size as low resolution To optimize computational efficiency, the time axis reconstruction formulas are all performed under low-resolution features, as designed below: in, A queue of historical features can be used to depict the dynamic changes in the background. For storage Low-resolution features at any given moment; This represents the high-resolution features at that moment; This represents the spatial difference characteristics at that moment; This involves transforming spatial difference features into low-resolution features using the Pixel_Shuffle operation. Furthermore, Low-resolution features of the current frame In the The value at each position corresponds to a The eigenvectors; similarly For low-resolution features in the historical queue in the first The value at each position corresponds to a eigenvectors, Spatial difference features in historical queues In the The value at each position corresponds to a eigenvectors; and These are transformation matrices for the query variable and the key variable, both of which are The transformation matrix; It is a value transformation matrix, which is The transformation matrix is ​​then used to form the attention matrix, as follows: in, yes The transpose of the given form is then normalized using the Softmax operation: Indicates position i This characteristic is consistent with the historical queue. The probability value of generating similarity values ​​for features at the same location at the same time, where, It is an exponential function; the background information recovered from historical information is as follows: This recovered spatial difference feature Then, through the Pixel_Shuffle operation, the spatial difference features are transformed into those for high-resolution reconstruction as follows: The temporal difference between the reconstructed spatial difference features and the spatial difference features of the current frame is as follows: in, This is due to time differences.

[0028] Figure 3 It is a dynamic queue maintenance and update graph, updating the dynamic background queue. A new background image is calculated based on the current frame and the foreign object segmentation result. For this new background image, the backbone network is used to calculate its corresponding features and replace the oldest feature in the feature queue to obtain a new feature queue.

[0029] 1) For each new image frame input, a new background is generated based on the monitoring results, as follows: in, It is a smoothing factor, usually chosen as 0.95. The background image stored from the previous moment is an image of the same size as the input image frame. It's a pixel-by-pixel multiplication operation. The binary image corresponding to the size of the original image is the foreign object monitoring result generated in step 4.

[0030] 2) For this new context, the backbone network is used to calculate its corresponding features, and the oldest feature in the feature queue is replaced to obtain a new feature queue, represented as: in, Stored in the history queue Background image of the time stamp. In the historical queue Low-resolution features of time ticks High-resolution features at the same time point, This represents the spatial difference characteristics at that moment. Features are spatial difference features after undergoing the Pixel_Unshuffle operation.

[0031] Figure 4 This is a flowchart of a text-driven railway surveillance video generation process, used to augment raw data and improve the generalization ability of the monitoring network. The input is the target railway video and the edited prompt word (Target prompt). The input video undergoes a preprocessing module to obtain a video edge map, which enhances the fidelity of the generated video. Initial noise values ​​are obtained through DDIM Inversion. The obtained noise is then sampled using DDIM, and the edge map and the target prompt are input into the ControlNet video diffusion module. Finally, the augmented new video is output.

[0032] Figure 5 It is based on Figure 4 The railway surveillance videos generated during the process are used to augment the original data by constructing different types of foreign object intrusion video datasets and annotation information, which are then used to train the foreign object detection network. Figure 5 The video demonstrates the results of video transformations achieved by altering weather conditions (e.g., rainy or snowy), seasons (e.g., spring), and time of day (e.g., day or night). These targeted video generation techniques can improve the network's ability to detect foreign objects in rainy or snowy weather, complex scenes, sudden changes in lighting, and dynamic backgrounds, thereby ensuring safer railway lines and train operations.

[0033] In addition, a quantitative evaluation comparison is provided, with the evaluation index being detection accuracy (F1 score), which is defined using the following indicators: Number of pixels correctly identified as foreground pixels -- (TP) Number of pixels correctly identified as background -- (TN) Background pixels incorrectly identified as foreground pixels -- (FP) Number of foreground pixels incorrectly identified as background -- (FN) Precision = TP / (TP+FP); Recall rate = TP / (TP + FN) The detection index is defined as follows: Precision (F1 score) = 2 * (precision * recall) / (precision + recall) As shown in Table 1, by using 20% ​​labeled videos and 80% synthetic videos, compared to using 100% labeled videos, the F1 score improved from 0.856 to 0.909. At the same time, the number of real videos was reduced by 80%, meaning that only 20% of the original labeled videos were needed to achieve improved monitoring results and reduce the difficulty of labeling.

[0034] Table 1: Quantitative comparison of the application prospects detection results of this invention on a railway surveillance video set constructed from mixed videos (data generated from 1000 video segments: original data 2:8) and directly trained from the original data. Table 2: Video text prompts generated by this invention, and the amount of data generated. Table 2 summarizes the prompts used for augmentation of the original videos and their corresponding proportions. In actual operation, to accelerate the training process, the original videos are randomly augmented in advance, and then the augmented railway surveillance videos are sampled based on the results of the validation set during the validation phase. Table 2 lists the prompts used for augmentation and their corresponding proportions.

Claims

1. A method for generating and monitoring railway foreign object videos based on text-driven and feature-based collaboration, characterized in that... Includes the following steps: Step 1: Spatial attention mechanism for low-resolution driving high-resolution reconstruction: Input a sequence of railway monitoring video frames, and focus on the current frame image... Low-resolution features are obtained through the backbone network. Features with high resolution ; Use low-resolution features to drive high-resolution features for background reconstruction and calculate the spatial difference with the features of the current frame. ,in, It is the number of channels. It is the width of the current frame image. It is the height of the current frame image. It is the field of real numbers; Step 2, Temporal Attention Mechanism for Low-Resolution-Driven High-Resolution Reconstruction: Based on the stored historical feature queue of railway surveillance videos, the low-resolution features obtained in Step 1 are used to drive the high-resolution features and perform temporal reconstruction, recovering spatial differences from the temporal axis, and calculating the spatial difference with the features of the current frame. The difference in time of acquisition ; Step 3: Calculate and optimize the probability map of foreign object intrusion limit: address the time difference generated in Step 2. The image is enlarged to its original size using bilinear interpolation, and then... The convolution combined with the Sigmoid function yields the foreign object probability map. Then, binarization is performed, followed by morphological post-processing with 3×3 erosion and 5×5 dilation operations to remove isolated noise and restore the integrity of the foreign object's outline, resulting in the final foreign object segmentation result, which is a binary image of the same size as the original image. ; Step 4, Dynamic Queue Update and Noise Suppression: Update the dynamic background queue. Calculate a new background image using the current frame and the foreign object segmentation result. For this new background image, use the backbone network to calculate its corresponding features and replace the oldest feature in the feature queue to obtain a new feature queue. Step 5: Network Training and Text-Based Video Augmentation: For the generated foreign object probability map, cross-entropy loss is used to calculate the loss between the foreign object probability map and the true value; at the same time, validation is performed on the validation set, and failure scenarios under common weather conditions are statistically analyzed. Based on the target text description constructed from the failure scenarios and the original video sequence, videos of foreign objects on railways are generated through the ControlNet module to augment the existing dataset and serve as augmented samples for subsequent training, thereby improving the adaptability of the samples to the working conditions.

2. The method for generating and monitoring foreign object videos on railways according to claim 1, characterized in that: Step 1 is as follows: Input a sequence of railway surveillance video frames, targeting the current frame image. Low-resolution features are obtained through the backbone network. Features with high resolution ; Use low-resolution features to drive high-resolution features for background reconstruction and calculate the difference with the current frame features; Use the Pixel_Shuffle operation to transfer high-resolution features Converted to the same spatial size as low resolution To optimize computational efficiency, all reconstruction formulas are performed using low-resolution features, as designed below: Low-resolution features of the current frame In the The value at each position corresponds to a eigenvectors; Low-resolution features of the current frame In the The value at each position corresponds to a eigenvectors; Features In the The value at each position corresponds to a eigenvectors; and These are transformation matrices for the query variable and the key variable, both of which are The transformation matrix; It is a value transformation matrix, which is The transformation matrix is ​​then used to form the attention matrix, as follows: in, Features The transpose of the given form is then normalized using the Softmax operation: in, It is an exponential function, and the background information recovered from other location information is as follows: This recovered background feature Then, through the Pixel_Shuffle operation, it is transformed into high-resolution reconstructed background features: in The reconstructed background features have the following spatial differences from the current frame features: 。 3. The method for generating and monitoring foreign object videos on railways according to claim 1, characterized in that: Step 2 is as follows: Based on the stored historical feature queue of railway surveillance videos, the low-resolution features obtained in step 1 are used to drive the high-resolution features for temporal reconstruction, and the spatial difference features with the current frame are calculated. The differences were used to obtain time difference characteristics. Spatial differences in step 1 Converted to the same spatial size as low resolution To optimize computational efficiency, the time axis reconstruction formulas are all performed under low-resolution features, as designed below: in, A queue of historical features can be used to depict the dynamic changes in the background. For storage Low-resolution features at any given moment; This represents the high-resolution features at that moment; This represents the spatial difference characteristics at that moment; To transform spatial difference features into low-resolution features using the Pixel_Shuffle operation; in addition, Low-resolution features of the current frame In the The value at each position corresponds to a The eigenvectors; similarly For low-resolution features in the historical queue in the first The value at each position corresponds to a eigenvectors, Spatial difference features in historical queues In the The value at each position corresponds to a eigenvectors; and These are transformation matrices for the query variable and the key variable, both of which are The transformation matrix; It is a value transformation matrix, which is The transformation matrix is ​​then used to form the attention matrix, as follows: in, yes The transpose of the given form is then normalized using the Softmax operation: Indicates position i This characteristic is consistent with the historical queue. The probability value of generating similarity values ​​for features at the same location at the same time, where, It is an exponential function; the background information recovered from historical information is as follows: This recovered spatial difference feature Then, through the Pixel_Shuffle operation, the spatial difference features are transformed into those for high-resolution reconstruction as follows: The temporal difference between the reconstructed spatial difference features and the spatial difference features of the current frame is as follows: in, This is due to time differences.

4. The method for generating and monitoring foreign object videos on railways according to claim 1, characterized in that: Step 4, dynamic queue update and noise suppression, achieves its purpose by updating the dynamic background queue. Specifically, for each new input image frame, a new background is generated based on the monitoring results, as follows: in, It is a smoothing factor, usually chosen as 0.

95. The background image stored from the previous time step is the same size as the input image frame. It's a pixel-by-pixel multiplication operation. The binary image corresponding to the size of the original image is the foreign object monitoring result generated in step 4; To address this new context, a backbone network is used to compute its corresponding features, and the oldest feature in the feature queue is replaced to obtain a new feature queue, represented as: in, Stored in the history queue Background image of the time stamp. In the historical queue Low-resolution features of time ticks High-resolution features at the same time point, This represents the spatial difference characteristics at that moment. Features are spatial difference features after undergoing the Pixel_Unshuffle operation.

5. The method for generating and monitoring foreign object videos on railways according to claim 1, characterized in that: In step 5, cross-entropy loss is used to calculate the foreign object probability map and the true value for the generated foreign object probability map. The losses between them are as follows: in, It is cross-entropy loss.

Images