A railway scene target detection and behavior recognition method based on spatiotemporal dual-stream features

Abstract

This application belongs to the field of railway engineering safety monitoring and computer vision technology, and discloses a method for target detection and behavior recognition in railway scenes based on spatiotemporal dual-stream features. The method includes: acquiring and preprocessing video data into image frame sequences; inputting the sequences into an improved target detection network (Mamba-Yolov11), which fuses local spatial features and global temporal context information by setting convolutional neural network paths and state-space model paths in parallel, and uses an improved C3k2_UIB module to achieve dynamic path selection, improving parameter efficiency; the network training uses the Focaler-IoU loss function to solve the problem of imbalanced training for small and difficult samples; for complex behaviors, the detected target regions are fed into the MILA-SF behavior recognition network, which efficiently identifies spatiotemporal behaviors through a fast and slow dual-path design. This invention can significantly improve the detection accuracy of targets such as wearable equipment and work tools in railway scenes.

Description

Technical Field

[0001] This invention relates to the fields of railway engineering safety monitoring and computer vision technology, and in particular to a method for target detection and behavior recognition in railway scenes based on spatiotemporal dual-stream features. Background Technology

[0002] The railway operating environment is complex and highly dangerous. Relying on manual inspections or traditional video surveillance for safety management suffers from low efficiency, poor real-time performance, and high missed detection rates. Therefore, developing automated and intelligent safety detection technologies has become an urgent need for the industry. However, existing technologies face the following bottlenecks when applied to railway scenarios:

[0003] First, in the field of object detection, mainstream convolutional neural network (CNN)-based algorithms, such as the YOLO series, have inherent limitations in their core convolutional operations due to the local receptive field. This results in insufficient ability to handle long-distance dependencies common in railway scenes; for example, it is difficult to correlate distant people and equipment, or it is prone to false positives and false negatives when the background is cluttered or multiple objects are stacked and occluded. The Transformer architecture based on self-attention mechanisms, introduced to address this issue, enhances global modeling capabilities, but its quadratic computational complexity leads to huge memory and computing power overhead when processing high-resolution surveillance videos, making it unsuitable for resource-constrained deployments of edge devices in railway sites. Furthermore, its patching method may disrupt the spatial continuity of images, adversely affecting the detection performance of small targets such as safety helmets and tools.

[0004] Secondly, in the field of behavior recognition, traditional methods typically use continuous, complete video frames as input. These methods suffer from computational redundancy and background interference; the large amount of static background information in the video not only consumes valuable computational resources but may also interfere with the model's learning of key behavioral features, especially in complex maintenance depot environments with large equipment such as overhead cranes and locomotives. Furthermore, some dual-stream networks that rely on optical flow computation to capture motion information experience significant time consumption in their optical flow extraction process, making it difficult to meet the real-time requirements for instantaneous response to violations in railway operations.

[0005] Finally, the data from railway operation scenarios presents unique challenges. On the one hand, many critical operational behaviors, such as raising the pantograph and spreading sand, are highly time-series, and single-frame image information is insufficient for accurate judgment, requiring models to have efficient temporal information modeling capabilities. On the other hand, the data sample distribution is extremely uneven. For example, the sample size of high-frequency common targets such as "work clothes" is huge, while the samples of critical dangerous behaviors such as "drilling into the car" and "limbs entering gaps" are extremely scarce. This causes the gradient of traditional models to be dominated by high-frequency samples during training, resulting in poor performance in detecting low-frequency but critical abnormal events.

[0006] Therefore, there is an urgent need for a new intelligent detection method for railway scenes that can take into account both global context and local details, achieve high accuracy while maintaining computational efficiency, and effectively handle temporal information and sample imbalance problems. Summary of the Invention

[0007] Based on this, the present invention provides a railway scene target detection and behavior recognition method based on spatiotemporal dual-stream features. It aims to take into account both global context and local details, achieve high accuracy while maintaining computational efficiency, and effectively handle the problems of temporal information and sample imbalance.

[0008] In a first aspect, the present invention provides a method for target detection and behavior recognition in railway scenes based on spatiotemporal dual-stream features, comprising the following steps:

[0009] Acquire video data of a railway scene and preprocess the video data into a sequence of image frames with a preset temporal relationship;

[0010] The image frame sequence is input into a pre-trained target detection network. The target detection network adopts an architecture that includes a backbone network, a neck network, and a head network. The backbone network sets up a convolutional neural network path and a state space model path in parallel to extract local spatial features and global temporal context information of the image frame sequence, respectively. The neck network performs multi-scale fusion of the local spatial features and the global temporal context information. Finally, the head network outputs the category information and location information of one or more targets in the image frame.

[0011] Based on the target location information output by the target detection network, a region image sequence containing the target is extracted from the image frame sequence;

[0012] The region image sequence is input into a MILA-SF behavior recognition network. The MILA-SF behavior recognition network includes a slow path and a fast path. The slow path processes the input region image sequence at a low frame rate and a high number of channels to capture spatial semantic information. The fast path processes the sequence at a high frame rate and a low number of channels to capture temporal dynamic information. The target's work behavior recognition result is finally output through feature fusion between the slow path and the fast path.

[0013] As an optional implementation of the first aspect of this application, the backbone network specifically includes: extracting basic features from the input image frame sequence through an initial convolutional layer; feeding the extracted basic features into at least one improved C3k2_UIB module for cross-stage feature fusion and dynamic path selection; inputting the features output by the C3k2_UIB module in parallel to the next layer of the convolutional neural network path and the Mamba module of the state space model path, wherein the Mamba module processes long sequence dependencies through a selective state space model; performing multi-scale feature aggregation on the output features of the improved C3k2_UIB module and the output features of the Mamba module through a spatial pyramid pooling module; and performing cross-channel attention operation on the output features of the spatial pyramid pooling module through a cross-stage partial similarity attention module.

[0014] As an optional implementation of the first aspect of this application, the processing of the improved C3k2_UIB module includes: receiving input features, passing them through a convolutional layer, and splitting the output features into two paths; sending one of the features into a UIBBlock for processing, wherein the processing of the UIBBlock includes: extracting low-frequency structural features through an initial depth convolution; dynamically increasing the number of channels according to a preset expansion ratio through a 1x1 extended convolution to enhance feature representation capability; capturing multi-scale detail features in high-dimensional space through an intermediate depth convolution; compressing channels through a 1x1 projective convolution to fuse channel information; fusing the output features of the UIBBlock with the other unprocessed feature, and sending the fused features into a CBS module composed of convolution, batch normalization, and activation functions for final processing.

[0015] As an optional implementation of the first aspect of this application, the processing of the Mamba module includes: splitting the input sequence into an SSM branch input and a gated branch input through linear projection; sequentially performing depthwise convolution and SiLU activation function processing on the SSM branch input, and then feeding it into a bidirectional selective scanning structure, which transitions the state based on dynamically generated parameters to capture contextual information; performing element-wise gated multiplication between the output of the bidirectional selective scanning structure and the gated branch input processed by the activation function; and finally outputting the sequence after temporal information modeling and feature modulation through a projection layer.

[0016] As an optional implementation manner of the first aspect of the present application, the neck network of the target detection network adopts a pyramid feature fusion architecture, and its processing process includes: magnifying the size of the deep high-semantic feature map output by the backbone network through an upsampling operation; through a feature splicing operation, fusing the upsampled high-level features, the local spatial features output by the convolutional neural network path in the backbone network, and the global temporal context information output by the state space model path in the backbone network to construct a feature pyramid containing rich details, semantics and temporal logic.

[0017] As an optional implementation manner of the first aspect of the present application, the method further includes: the target detection network is trained using the Focaler-IoU loss function, and the calculation process of the Focaler-IoU loss function includes: setting a lower threshold d and an upper threshold u, where 0 ≤ d < u ≤ 1; using a linear interval mapping method to reconstruct the original intersection over union (IoU) value between the detection box and the ground truth box into a new IoU value , and the calculation formula is: , to filter out extremely difficult samples with an IoU value less than d and easy samples with an IoU value greater than u; based on the new IoU value , calculate the final Focaler-IoU loss , and the calculation formula is: .

[0018] As an optional implementation manner of the first aspect of the present application, the method further includes: preprocessing the video data: unifying the video stream into the mp4 format with a resolution of 1080×720 and a frame rate of 30fps; sampling and outputting a static image frame from the video at an interval of every 2 seconds to form the image frame sequence; using the LabelImg tool to perform target annotation on each static image frame to generate an annotation file conforming to the YOLO standard format, and the YOLO standard format includes the class index of the target, the normalized center point x coordinate, the center point y coordinate, the width and the height.

[0019] As an optional implementation manner of the first aspect of the present application, in the MILA-SF behavior recognition network, feature fusion is performed through a lateral connection provided between the slow path and the fast path, and the lateral connection specifically includes: performing a time dimension downsampling and a channel dimension upsampling transformation on the features output by the fast path to match the feature dimension of the slow path; and fusing the transformed fast path features with the features of the slow path.

[0020] In a second aspect, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0021] Thirdly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. Object Detection Network (Mamba-Yolov11): By introducing the Mamba state-space model in parallel into the Yolov11 architecture, it combines the local perception capability of CNNs with the global context modeling capability of SSM. SSM handles long sequence dependencies with linear complexity, solving the problems of limited receptive field and high computational cost of attention mechanism in traditional CNNs, and effectively improving the detection performance of long-distance, occluded and small targets.

[0024] 2. Temporal and Spatial Feature Fusion Detection: The Neck network fuses local spatial features from the CNN path in the backbone network with global temporal features from the Mamba path across layers, enabling the network to make judgments based on the logical relationships between consecutive frames, thus improving the accuracy of target recognition. Simultaneously, the improved C3k2_UIB module adaptively processes high- and low-frequency information through conditional branching and dynamic path selection, improving parameter efficiency.

[0025] 3. Focaler-IoU loss function solves the problem of imbalanced samples: In response to the problem of scarce samples for some categories in railway scenarios, the innovative Focaler-IoU loss function sets an IoU interval, which enables the model to focus on learning from samples of moderate difficulty. Combined with dynamic weighting, it significantly improves the detection accuracy of small sample categories such as "head" and "overhead crane hoisting".

[0026] 4. MILA-SF Complex Behavior Recognition: By using target detection results as input for behavior recognition, the computational load is significantly reduced. The MILA-SF network draws on the SlowFast concept, capturing temporal dynamics and spatial semantics through fast and slow dual paths respectively, and replacing traditional convolution with MILA modules (fused with SSM), achieving efficient and accurate recognition of complex long-term temporal behaviors such as "buffer disassembly and assembly" and "drilling rig". Attached Figure Description

[0027] Figure 1 This is a flowchart of a railway scene target detection and behavior recognition method based on spatiotemporal dual-stream features according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the overall architecture of the Mamba-Yolov11 target detection network in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the internal structure of the improved C3k2_UIB module in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the internal structure and data flow of the Mamba module in an embodiment of the present invention;

[0031] Figure 5 This is a comparative evaluation chart of the detection performance of the "head" category before and after using the Focaler-IoU loss function in this embodiment of the invention;

[0032] Figure 6 This is a schematic diagram of the overall architecture of the MILA-SF complex behavior recognition network in an embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the internal structure of the MILA module in an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0036] Example 1

[0037] Please see Figure 1 This is a flowchart illustrating a railway scene target detection and behavior recognition method based on spatiotemporal dual-stream features, provided by an embodiment of the present invention. The method may include the following steps:

[0038] S1: Acquire video data of the railway scene and preprocess the video data into a sequence of image frames with a preset temporal relationship.

[0039] This step primarily transforms the raw surveillance video into a structured dataset. First, the videos are standardized by unifying their size to mp4 format, resolution to 1080×720, frame rate to 30fps, and duration to 40 minutes. They are then renamed according to a fixed rule, such as video1, video2, etc. Next, video processing tools are used to extract consecutive static frames from each video segment at 2-second intervals, generating 1200 images (formula: 40min×60s / min÷2s / frame = 1200 images). These images are then named sequentially from video1_1.jpg to video1_1200.jpg. Finally, image annotation and format conversion are performed using the LabelImg tool, with annotations conforming to the YOLO standard. The annotation categories cover key elements of railway operations: yellow hats, blue hats, white hats, red hats, red T-shirts, work clothes, heads, overhead cranes, factory motor vehicles, overhead cranes, tool bags, buckets, warehouse doors, buffers, ladders, lifting frames, trenches, locomotives, fire extinguishers, etc. Each image generates a corresponding .txt annotation file, storing the corresponding target's category and normalized coordinate information. The dataset is divided into training, validation, and test sets in a 7:2:1 ratio. During training, images undergo random rotation, scaling, and horizontal flipping to enhance the model's generalization ability.

[0040] S2: The image frame sequence is input into a pre-trained target detection network. The target detection network adopts an architecture including a backbone network, a neck network, and a head network. The backbone network sets up a convolutional neural network path and a state space model path in parallel to extract local spatial features and global temporal context information of the image frame sequence, respectively. The neck network performs multi-scale fusion of the local spatial features and the global temporal context information. Finally, the head network outputs the category information and location information of one or more targets in the image frame.

[0041] In this embodiment, the object detection network (Mamba-Yolov11) is an improved and innovative design based on Mamba and Yolov11. Traditional object detection algorithms such as YOLOv5 and YOlov8 often suffer from problems such as the locality of convolutional layers, resulting in weak modeling ability for long-distance object associations, such as individuals in dense crowds, lack of global perception mechanisms, and high false detection rates in complex scenes. Attention-based architectures such as ViT or SwinTransformer often suffer from the secondary complexity of self-attention, leading to a surge in memory consumption at high-resolution inputs, which is detrimental to lightweight deployment designs. ViT's patching disrupts spatial continuity, degrading performance in small object detection. Mamba-Yolov11, while lightweighting the Yolov11 network architecture, uses the core operator (SSM) in Mamba to transmit global temporal context information through hidden states, solving the problem of limited receptive fields in traditional CNNs. Simultaneously, it extracts local spatial features through the convolutional neural network paths in the Yolov11 network, compensating for the SSM's deficiency in perceiving local image structures. Mamba, as a selective state-space model, does indeed demonstrate significant advantages in processing time-series data, particularly outperforming traditional models in long sequence modeling, computational efficiency, and dynamic pattern capture. In object detection, temporal features are incorporated, recognizing that accurate object identification cannot be achieved from a single image alone and requires multi-angle assessment. Continuous images in the time domain are used for multi-angle evaluation, thereby improving the accuracy of object detection.

[0042] like Figure 2 As shown, the object detection network (Mamba-Yolov11) adopts the classic YOLO hierarchical architecture of Backbone-Neck-Head. The Backbone's dual-stream parallel computation integrates the convolutional neural network path and the state-space model path. The specific data flow and processing within its architecture are as follows:

[0043] Images of different sizes are first preprocessed to unify their size to (3, 640, 640). They are then fed into the spatial channel of the dual-stream channel. The first layer undergoes a 3x3 convolution to output (16, 320, 320). This output is then fed into the first layer convolution to obtain a (32, 160, 160) matrix. This matrix is ​​then fed into the second layer C3k2_UIB module to obtain a (64, 160, 160) matrix. This matrix is ​​then fed into the third layer convolution module to obtain a (64, 80, 80) matrix. Finally, this matrix is ​​fed into the fourth layer C3k2_UIB module to obtain a (128, 80, ...) matrix. The 80) matrix output is fed into the 5th layer convolutional module to obtain a matrix of (128, 40, 40). Its output is then fed into the 6th layer C3k2_UIB module to obtain another matrix of (128, 40, 40). This is followed by the 7th layer convolutional module to obtain a matrix of (256, 20, 20). This is then fed into the 8th layer C3k2_UIB module to obtain yet another matrix of (256, 20, 20). Finally, it is fed into the 9th layer spatial pyramid pooling module to obtain another matrix of (256, 20, 20). This is then fed into the cross-stage partial similarity attention module to obtain yet another matrix of (256, 20, 20). In the dual-stream channel, the 11th layer convolutional input in the temporal channel is the output of the 0th layer convolution. After 11 layers of convolution, the output is a matrix of (32, 160, 160), which is then fed into the 12th layer convolutional module to obtain an output of (64, 80, 80). Finally, it is fed into the 13th layer Mamba module, where the output matrix size remains unchanged across all channels. Then, the matrix is ​​fed into the convolutional layer of layer 14 to obtain (128, 40, 40), and then into the C3k2_UIB module of layer 15 to reduce the number of channels and obtain a (64, 40, 40) matrix output. The output matrix of the Mamba module of layer 16 has the same size and number of channels. Then, the matrix is ​​fed into the C3k2_UIB module of layer 17 to obtain a (64, 40, 40) matrix output. The output matrix of the Mamba module of layer 18 has the same size and number of channels. The matrix is ​​fed into the convolutional module of layer 19 to obtain a (128, 20, 20) matrix. The output matrix of the Mamba module of layer 20 has the same size and number of channels. Then, the matrix is ​​fed into the spatial pyramid pooling module of layer 21 to obtain (128, 20, 20), and then fed into the cross-stage partial similarity attention module to obtain a (128, 20, 20) matrix.Then, feature fusion is performed in the Neck network. The outputs of layer 10 (256, 20, 20) and layer 22 (128, 20, 20) are concatenated to obtain the output matrix (384, 20, 20). This output is then fed into layer 24 for upsampling to obtain (384, 20, 20). The outputs of layer 6 (128, 40, 40), layer 16 (64, 40, 40), and layer 24 (upsampled) (384, 40, 40) are concatenated to obtain the output matrix (576, 40, 40). This output is then fed into layer 26 (C3k2_UIB module) to obtain the matrix (192, 80, 80). Finally, the outputs of layer 4 (128, 80, 80), layer 13 (64, 80, 80), and layer 27 (upsampled) (192, 80, 80) are concatenated to obtain the output matrix (384, 80). The output of (80) is then fed to the 29th layer C3k2_UIB module to obtain the (128, 80, 80) matrix; it is then fed into the 30th layer convolution to obtain the output (64, 40, 40). The output of (64, 40, 40) from the 18th layer, the 26th layer output (192, 40, 40), and the 30th layer convolution (64, 40, 40) are concatenated to obtain the output matrix (320, 40, 40). This matrix is ​​then fed into the 32nd layer C3k2_UIB module to obtain the (128, 40, 40) matrix output, fed into the 33rd layer convolution to reduce its size to obtain (128, 20, 20), fed into the 34th layer to concatenate the (384, 20, 20) from the 23rd layer, and output to obtain (512, 20, 20); finally, it is fed into the 35th layer C3k2_UIB module to reduce the number of channels to (256, 20, 20). Layers 29, 32, and 35 are used as input for detection.

[0044] Understandably, the backbone first extracts basic features layer by layer from the input image. Convolutional layers (Conv) extract these basic features by stacking standard convolutions, gradually reducing the feature map size and increasing the number of channels. The improved C3k2_UIB module, based on the CSP concept, introduces cross-stage feature fusion through residual connections, concatenated convolutions, batch normalization, and activation functions, enhancing gradient flow and feature diversity. The Mamba module, inserted into the backbone, handles long sequence dependencies with linear complexity, enhancing the ability to model global context while maintaining the local perceptual advantages of CNNs. The Spatial Pyramid Pooling (SPPF) module aggregates features from different receptive fields through parallel multi-scale pooling, enhancing the robustness of feature maps to the target scale while avoiding the spatial dimensionality expansion problem of traditional SPP. The Cross-Stage Partial Similarity Attention (C2PSA) module, a cross-channel attention mechanism, focuses on the correlation between local details and global structure through pixel-level similarity calculation and partial alignment, improving the discriminative power of feature representations. The Backbone output is fused through multiple paths, such as the Mamba branch at layer 13, and spliced ​​with the main chain to form multi-layer heterogeneous features, achieving an increasing number of channels and a decreasing size, providing a multi-scale foundation for the subsequent Neck.

[0045] Secondly, Neck employs a pyramid feature fusion architecture, integrating the multi-layer outputs of the backbone network through upsampling and feature concatenation to construct a rich feature pyramid ranging from shallow details to deep semantics. Lateral connections, such as in layer 25, concatenate the improved C3k2_UIB module output with the upsampled high-level features and Mamba temporal features, fusing shallow textures (edges, colors) with deep semantics (category, structure) and temporal logic. Cross-layer interaction between Mamba and CNN: the outputs of the 18th layer Mamba branch and convolutional outputs are fused and input into the next layer, the 20th layer Mamba module. From bottom to top, the number of channels in the feature maps gradually decreases, from high resolution with low channels to low resolution with high channels, adapting to different task requirements.

[0046] Finally, the Head, based on the multi-scale features output by the Neck, designs independent prediction branches for specific tasks, supports multi-scale prediction, and outputs the category and location information of the target in the image frame; the figure marks multiple Heads, which respectively handle the prediction of different scales. Large targets use low-resolution high semantic features, while small targets use high-resolution detail features.

[0047] It's important to note that traditional behavior recognition training methods employ the strategy of inputting an entire image or a series of consecutive frames. While this preserves all visual information from the image, it also introduces numerous drawbacks. From a computational resource consumption perspective, the massive amount of input data necessitates the use of numerous computing units during both feature extraction and subsequent model training. This significantly increases training time and places extremely high demands on hardware computing power, limiting the algorithm's application in resource-constrained scenarios. In contrast, the dual-stream temporal spatial feature-based behavior recognition technology extracts regions by utilizing the target bounding boxes output from object detection, incorporating Mamba temporal features to demonstrate significant technical advantages. In terms of computational efficiency, this approach drastically reduces the amount of data input to the model, focusing only on the local region where the target behavior occurs and the temporal sequence of the corresponding consecutive images, resulting in a significant reduction in computational costs and a substantial shortening of training time. Furthermore, in terms of spatial feature extraction, it proposes a region-focusing strategy integrating the C3K2_UIB module. Through multiple optimizations, it achieves efficient and accurate behavior recognition, including dynamic frequency adaptation: automatically selecting high-frequency / low-frequency processing paths through a conditional branch decision mechanism to suppress background noise interference; and small target enhancement: initial depthwise convolution captures the global structure, while intermediate convolutions enhance local details. Parameter quantity from Down to This reduces the computational burden in the behavior recognition input stage.

[0048] like Figure 3 As shown, in the YOLO architecture, the improved C3k2_UIB module is one of the key modules for feature fusion. The present invention improves the performance of the C3k2 module by the following improvements: first, a convolution (Conv) is fed in, and then the output features are split (Split) and then fed into UIBBlock.

[0049] Compared to the traditional C3k2 module's fixed-order standard convolution stacking and fixed channel dimension, the improved C3k2_UIB implements conditional branching and dynamic path selection, primarily using depthwise separable convolutions with implicit frequency adaptation. This achieves adaptive computation paths, improving parameter efficiency and suppressing high frequencies while enhancing low frequencies. A dynamic channel expansion mechanism adjusts the number of channels dynamically based on the expansion ratio.

[0050] Specifically, the UIBBlock processing procedure includes the following steps:

[0051] (1) Starting depthwise convolution (Start_dw)

[0052] A conditional branching decision mechanism is introduced to avoid redundant computation and dynamically adapt to feature complexity. The number of parameters is reduced from... become The formula for calculating the initial depthwise convolution (Start_dw) is as follows:

[0053]

[0054] In the formula, C represents the channel index (each channel is calculated independently), K represents the kernel size, and Ss represents the stride, which is controlled by middle_dw_downsample. This represents the initial depth of the convolution weights. Pi represents the padding value in the input tensor. Spatial filtering is performed to extract low-frequency structural features without changing the number of channels. l This represents the spatial coordinates of the output feature map.

[0055] (2) 1×1 Expanded Convolution (Expand_Conv)

[0056]

[0057] In the formula, For the input feature map, For convolution weights, To output the feature map, Given a weight matrix shape, the number of input channels C is dynamically expanded according to the expansion ratio r through a 1x1 convolution. =make_divisible(C×r,8): Dynamically expands the number of channels (expand_ratio=r), increasing the dimensionality to... This enhances the ability to represent features. The expansion ratio r controls the balance between model capacity and computational cost.

[0058] (3) Middle-depth convolution (Middle_dw)

[0059]

[0060] In the formula, Km represents the convolution kernel size (middle_dw_kernel_size). The stride is determined by `middle_dw_downsample`, which performs secondary spatial filtering in the expanded high-dimensional space to capture multi-scale details (such as texture and edges). `Pi` represents the padding value in the input tensor. Downsampling control: if `middle_dw_downsample` = 1, then `Sm` = stride; otherwise, `Sm` = 1, flexibly adapting to different resolution requirements. This represents the depthwise convolution weights. This is to output the spatial coordinates of the feature map Y'.

[0061] (4) 1×1 projective convolution (Project_Conv)

[0062]

[0063] In the formula, O represents the output feature map. For projection weights, This indicates the number of output channels. Channels are compressed to the target dimension to reduce computation and fuse channel information. To expand the number of channels, it is usually greater than This approach decouples spatial and channel information, using depthwise convolution to process spatial information and 1×1 convolution to process channel information. Dynamic frequency response: the initial convolution with its large receptive field captures low frequencies, while intermediate convolutions provide local filtering to enhance high frequencies.

[0064] Parameter efficiency: By separating spatial / channel computation, the complexity is reduced from... Down to This process compresses and fuses the rich spatial features extracted by the previous depth convolution.

[0065] Finally, the features of the first and last UIBBlocks are fused and fed into the CBS module, which consists of 3×3 convolutional kernels. This module further fuses channel information, adjusts feature dimensions, performs batch normalization, standardizes feature distribution, and solves the internal covariate shift problem. The SiLU activation function is then used to enhance feature expressiveness.

[0066] like Figure 4 As shown, the inspiration for introducing the visual module Mamba lies in the temporal characteristics of continuous systems, and the key to these temporal characteristics is SSM, or Structured State-Space Sequence Model. Continuous systems utilize hidden states... Mapping a 1-D function or sequence → The system uses A∈ As an evolutionary parameter, , As a projection parameter, N represents the hidden state vector. The size of the dimension. The working principle of a continuous system is as follows:

[0067]

[0068] The Structured State-Space Sequence Model (SSM) is a discrete version of the Mamba continuous system, which includes a time-scale parameter Δ that transforms the continuous parameters A and B into discrete parameters. , A commonly used transformation method is the zero-order hold (ZOH), which is defined as follows:

[0069]

[0070]

[0071] After discretizing A and B, the discretized version using step size Δ can be rewritten as:

[0072]

[0073] The core process of module design lies in the module first processing the input feature map. Convert into a sequence through multiplication. The input sequence is split into two parallel branches through linear projection. and The formula is as follows:

[0074]

[0075] in For SSM branch input, For gated branch input, This is the projection matrix.

[0076] The branch is processed by depthwise convolution ( After activation with SiLU, the formula is as follows:

[0077]

[0078] In the above formula, and This represents pointwise convolution and depthwise convolution operations in deep learning.

[0079] It is then fed into a bidirectional selective scanning structure (forward SSM + reverse SSM).

[0080]

[0081] in This indicates a scan operation; the dynamically generated parameter Δ controls the state transition; the bidirectional output is the fused z-branch after activation. Gated multiplication is performed.

[0082] The gating formula is as follows:

[0083]

[0084] in This is the weight matrix;

[0085] Finally, the processed sequence is output through the projection layer, transforming the sequence into a feature map. The overall process achieves content-aware modeling through dynamic parameterized state transitions, captures global context using bidirectional scanning, and modulates feature responses using a gating mechanism, ultimately achieving efficient long sequence modeling while maintaining linear complexity.

[0086] In this step, the object detection network is trained using the Focaler-IoU loss function.

[0087] It is important to note that boundary box regression plays a crucial role in object detection, and the localization accuracy of object detection largely depends on the loss function of bounding box regression. Existing research improves regression performance by utilizing the geometric relationships between bounding boxes, while neglecting the impact of the distribution of easy and difficult samples on bounding box regression. In this invention, the influence of the distribution of easy and difficult samples on the regression results is analyzed, and then Focaler-IoU is proposed to improve the detector's performance in different detection tasks by focusing on different regression samples.

[0088] Focaler-LOU is a loss function used for bounding box regression in object detection. The basic principle of this method can be understood from the following aspects:

[0089] 1. Focusing on Different Regression Samples: Focaler-loU improves detector performance in different detection tasks by focusing on different regression samples. This is achieved by reconstructing the loU loss through linear interval mapping, thus focusing on different samples.

[0090] 2. Addressing the distribution of difficult and simple samples: It analyzes and considers the impact of the distribution of difficult and simple samples in bounding box regression on the regression results, which is an aspect often overlooked in the traditional IOU loss function.

[0091] 3. Improve existing bounding box regression methods: Focaler-loU uses its unique method to make up for the shortcomings of existing bounding box regression methods, thereby further improving detection performance in different detection tasks.

[0092] To focus on different regression samples and thus different detection tasks, this invention uses a linear interval mapping method to reconstruct the IoU loss, which allows for improved marginal regression. The formula is as follows:

[0093]

[0094] IoU is the original Intersection over Union (IoU) value, [d,u]∈[0,1]. d is the lower threshold, used to filter out overly difficult outlier samples to prevent the model from overemphasizing them. u is the upper threshold, used to filter out overly simple samples, allowing the model to focus on learning more valuable samples. By adjusting the values ​​of d and u, the IoU can be focused on different regression samples. Its loss is defined as follows:

[0095]

[0096] Finally The loss is applied to the existing IoU-based bounding box regression loss function.

[0097] like Figure 5 The figure shows a comparison evaluation of the detection performance of the "head" category before and after adopting the Focaler-IoU loss function in this embodiment. This evaluation figure only tested 6 types of labels and systematically demonstrated the multi-dimensional performance of the target detection model through two sets of 4 sub-figures: (1) F1-confidence curve, which reflects the F1 performance of the model under different confidence thresholds and reflects the stability of the model in distinguishing positive and negative samples. The horizontal axis is the prediction confidence and the vertical axis is the F1 score; (2) Precision-recall curve, which shows the trade-off between detection precision and coverage. The horizontal axis is the recall and the vertical axis is the precision. It can be seen from the figure that for data labels with a large number of samples, the loss function is not significantly optimized, but for small targets and targets with fewer sample data, the peak F1 score is greatly improved from 0.65 to 0.81. When the recall is 0.6, the precision before the loss function improvement is about 0.6, while the precision after the improvement is increased to 1.

[0098] S3: Based on the target location information output by the target detection network, extract the region image sequence containing the target from the image frame sequence.

[0099] S4: Input the region image sequence into a MILA-SF behavior recognition network. The MILA-SF behavior recognition network includes a slow path and a fast path. The slow path processes the input region image sequence with a low frame rate and a high number of channels to capture spatial semantic information. The fast path processes the sequence with a high frame rate and a low number of channels to capture temporal dynamic information. Through feature fusion between the slow path and the fast path, the target's work behavior recognition result is finally output.

[0100] For complex situations such as buffer disassembly and assembly, overhead crane operation and commissioning, and drilling rig operation in the maintenance depot, simply incorporating temporal features into object detection is far from sufficient; multi-frame incorporation is needed for better recognition of corresponding actions. First, an action recognition dataset is constructed based on the target location information output by the object detection network. The video is first edited into a 15-minute video, and then each 15-minute video is edited into images at 1 frame per second. Next, Faster R-CNN in Detectron2 is used to detect all images, and the detection results are converted into CSV files that VIA can recognize. Finally, the CSV file generated in the previous step is imported into VIA. After opening VIA and importing the images, actions are defined, separated by commas in English. After constructing the target region image dataset, it is used to train the subsequent MILA-SF action recognition network.

[0101] like Figure 6 As shown, the MILA-SF behavior recognition network structure is based on SlowFast, with input dimensions of [B, T, C, H, W].

[0102] Slow Pathway: Captures semantic information in video, such as hand gestures used by crane operators. Low frame rate sampling, but with a large number of channels, enables the network to learn complex, static features.

[0103] Fast Pathway: Captures motion information in video, such as a fleeting action like sand being sprinkled. High frame rate sampling with a small number of channels allows it to quickly and lightweightly handle subtle changes over time.

[0104] The two paths are connected laterally to integrate information, and the advantages of both are combined for classification.

[0105] B (Batch Size): The number of video segments processed simultaneously during a single training or inference iteration. For example, B=8 means processing 8 video segments at a time.

[0106] T (Temporal Length): The number of frames contained in each video segment. T differs between slow and fast paths. Slow paths have a low frame rate, extracting one frame every 16 frames from the video, so T_slow is 4 or 8. Fast paths have a high frame rate, extracting one frame every 2 frames from the video, so T_fast is 4 times T_slow, α=4.

[0107] C (Channels): The number of channels in each frame of the image. Usually, it is RGB three-channel C=3.

[0108] H (Height): The height of the frame image in pixels.

[0109] W (Width): The width of the frame image in pixels, referred to as S below.

[0110] In the design of the MILA-SF action recognition network, the input data is preprocessed to 224 pixels. Let T=64, 64 frames are continuously extracted from the original video. S=224: each frame is scaled to 224x224 pixels. The slow path has a stride of 16, 1², taking one frame every 16 frames from the 64 frames. 64 / 16 = 4. The input becomes: 4 × 224². The fast path has a stride of 2, 1², taking one frame every 2 frames from the 64 frames. 64 / 2 = 32. The input becomes: 32 × 224². The temporal resolution is 8 times that of the slow path (α = 32 / 4 = 8).

[0111] Initial convolution and pooling convolution, slow path: 1×7², 64. Convolution kernel: No convolution in the temporal dimension, only 7x7 convolution in the spatial dimension. Output channels: 64. Spatial size: Due to stride 2², 224 -> 112. Output size: 4× 112² (number of channels becomes 64). Fast path: 5×7², 8 convolution kernel: Convolution in the temporal dimension, 7x7 in the spatial dimension. Aggregates temporal information. Output channels: Only 8. This reflects the "narrow" design, number of channels β = 8 / 64 = 1 / 8. Output size: 32× 112².

[0112] Both paths use max pooling, halving the space size: 112 -> 56. Output size: slow path: 4 × 56², fast path: 32 × 56².

[0113] Figure 6 In the MILA-SF behavior recognition network, the main body consists of four MILA block stages (MILA1, MILA2, MILA4, MILA5), similar in structure to ResNet, but with important differences. Each block stage comprises multiple Mamba blocks. After the first MILA block, the input is 4×56² / 32×56² and the output is 4×56² / 32×56², with the spatial size unchanged; the first block uses stride=1. After the second MILA block, the input is 4×56² / 32×56² and the output is 4×28² / 32×28², with the spatial size halved, achieved by stride=2. After the third MILA block, the input is 4×28² / 32×28² and the output is 4×14² / 32×14², with the spatial size halved again. After the fourth MILA block, the input is 4×14² / 32×14² and the output is 4×7² / 32×7², with the spatial size finally halved. The time dimension T typically remains unchanged during the residual stage until the final global pooling.

[0114] Figure 6 The key link clearly shown in the diagram is the lateral connection, which occurs at the input of the MILA block stage. The purpose is to fuse the information of the fast path into the slow path, so that the slow branch knows what motion the fast branch has seen. (1) Operation upsampling / transformation: The feature map of the fast path (large T, small C) needs to be transformed to match the size of the slow path. (2) Time dimension: Since the T of the fast path is α times that of the slow path, it needs to be reduced to the same size as the slow path through time-dimensional pooling. (3) Channel dimension: The number of channels of the fast path is increased to the same size as the slow path through a 1x1x1 convolution. Fusion adds the transformed fast path features and slow path features element by element.

[0115] The final target's job behavior recognition result output: Global average pooling is performed on the final outputs of the two paths, 4×7² and 32×7², respectively. After pooling, the spatial dimensions H, W and the time dimension T are all flattened to 1. The output becomes two one-dimensional feature vectors: (1) Slow path: 1 × 1 × 2048 channels from the third MILA module; (2) Fast path: 1 × 1 × 256 channels from the fourth MILA module. In the concatenation layer, the two feature vectors are concatenated together: 2048 + 256 = 2294 dimensions. In the fully connected layer, this 2294-dimensional joint feature vector is mapped to the final category number output node through a fully connected layer to output the category number. In the Softmax and classification stage, the output is Softmaxed to obtain the probability of each category, and the behavior category with the highest probability is taken as the prediction result.

[0116] like Figure 7 As shown, in the MILA block, the input tensor shape is [B, T, C, S, S], where B is the number of samples, T is the time step, C is the number of channels, and S is the spatial dimension (height and width). The goal is to ensure that the final output shape is the same as the input, i.e., [B, T, C, S, S]. The components in the diagram include a Linear layer, a Linear Attention layer, a Conv layer, an activation function σ, and an SSM. The data flow starts from the input and passes through these components sequentially. To maintain the input and output shapes, each component needs to be designed not to change the tensor's dimensions (i.e., B, T, S, S remain constant). Only the number of channels C may change in the intermediate layers, but it is ultimately restored to the original C through the last Linear layer. The specific component design is as follows:

[0117] (1) Linear layer: Here it refers to pointwise convolution, that is, 1x1 convolution, which operates on the channel dimension. The kernel size is 1, the stride is 1, and the padding is 0. The number of output channels is set to be the same as the number of input channels to maintain the shape.

[0118] (2) Conv layer: refers to spatial convolutional layer, which operates on the spatial dimension S x S. Use a convolutional kernel (kernel_size) of 3 or other odd number of kernels, but ensure that the shape remains unchanged, stride=1, padding=s, and the number of output channels is set to be the same as the number of input channels to maintain the spatial size.

[0119] (3) Linear Attention Layer: This is an attention mechanism that typically handles sequence dimensions T. To preserve shape, the input tensor is first flattened to [B, T, C*S*S], then linear attention is applied to keep the output feature dimensions unchanged, and finally it is reshaped back to [B, T, C, S, S]. In this way, both the sequence length T and the feature dimensions remain unchanged.

[0120] (4) Activation function σ: ReLU is used, which does not change the shape of the tensor and the input and output are the same.

[0121] The following are the detailed input and output shapes for each component in the data stream. The tensor shape is always [B, T, C, S, S], where C is the number of input channels (unless the intermediate layer changes the number of channels, but C remains constant in this embodiment).

[0122] Table 1. Detailed Input / Output Shapes for Each Component Type

[0123]

[0124] As shown in Table 2, the method proposed in this invention (represented by the TSMY series for different scale models) uses TSMY, short for TwoStream MambaYOLO, which is the two-stream spatiotemporal MambaYolov11 object detection algorithm. This algorithm was compared with other mainstream algorithms on a self-made dataset. During the testing on the self-made dataset, the training phase employed diverse data augmentation operations such as random rotation, scaling, and horizontal flipping to effectively simulate the perspective changes, size differences, and lighting fluctuations that may occur in real-world scenes, significantly improving the model's generalization ability. In the final test results, despite the input images exhibiting different poses and proportions due to augmentation processing, such as the tilt angle of the target person and changes in the scaling of the background equipment, the model was still able to accurately detect key targets such as "yellow hat," "work shirt," and "white hat," and output corresponding confidence scores.

[0125] Table 2 Comparison experiments on the self-made dataset

[0126]

[0127] In Table 2, 'n', 's', 'm', 'l', and 'x' following 'TSMY' refer to different scale model variants built under the same MambaYolov11 network structure by adjusting dimensional factors such as depth and width. These different parameter counts correspond to different computational costs. 'TSMYn' represents an extremely lightweight nanoscale model, achieving an mAP50 of 91.9 with only 3.51M parameters and 5.24G FLOPs, significantly outperforming all listed YOLO versions in accuracy and demonstrating the efficiency advantages of the new architecture, making it an ideal choice for edge deployment. 'TSMYs' represents a balanced and practical lightweight model, with the number of parameters increased to 12.91M, but with further improved accuracy (mAP50: 92.4), providing excellent performance at a real-time speed of 18.7FPS, making it an excellent starting point for most real-time detection tasks. TSMYm represents the balance between performance and resource consumption. It offers higher accuracy (mAP50: 93.3), but significantly increases computational cost (123.62 GFLOPs) and latency (32.7 ms), making it suitable for scenarios with high accuracy requirements and sufficient computing power. TSMYl represents a large model aimed at high accuracy. It pushes accuracy to 94.2 with 38.68M parameters, but the inference speed drops to 9.1 FPS, serving server-side applications with extremely high accuracy requirements. TSMYx represents a cutting-edge model pursuing ultimate accuracy, boasting the largest number of parameters (86.7M) and the highest accuracy (mAP50: 95.0), but with the slowest inference speed (5.1 FPS). It is designed for tasks that prioritize extreme performance regardless of computational cost.

[0128] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a railway scene target detection and behavior recognition method based on spatiotemporal dual-stream features, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0129] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a railway scene target detection and behavior recognition method based on spatiotemporal dual-stream features, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0130] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0131] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0133] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for railway scene target detection and behavior recognition based on spatiotemporal dual-flow features, characterized in that, The method comprises the following steps: acquiring video data of a railway scene and preprocessing the video data into an image frame sequence with a preset timing relationship; inputting the image frame sequence into a pre-trained target detection network, the target detection network adopting an architecture comprising a backbone network, a neck network and a head network; the backbone network is provided in parallel with a convolutional neural network path and a state space model path to respectively extract local spatial features and global timing context information of the image frame sequence; the backbone network specifically comprises: performing basic feature extraction on the input image frame sequence through an initial convolutional layer; sending the extracted basic features to at least one improved C3k2_UIB module for cross-stage feature fusion and dynamic path selection; inputting the features output by the C3k2_UIB module in parallel to the next layer of the convolutional neural network path and a Mamba module of the state space model path, the Mamba module processing long sequence dependencies through a selective state space model; performing multi-scale feature aggregation on the output features of the improved C3k2_UIB module and the output features of the Mamba module through a spatial pyramid pooling module; performing cross-channel attention operation on the output features of the spatial pyramid pooling module through a cross-stage partial similarity attention module; the processing process of the improved C3k2_UIB module comprises: after receiving input features, a convolutional layer is used to divide the output features into two paths; one of the paths is sent to a UIBBlock for processing, the processing process of the UIBBlock comprising in sequence: extracting low-frequency structural features through an initial deep convolution; performing a 1x1 expansion convolution to dynamically increase the number of channels according to a preset expansion ratio to enhance the feature expression capability; capturing multi-scale detail features in a high-dimensional space through an intermediate deep convolution; compressing the channels through a 1x1 projection convolution to fuse channel information; fusing the output features of the UIBBlock with the other path of unprocessed features, and sending the fused features to a CBS module composed of a convolution, a batch normalization and an activation function for final processing; the neck network adopts a pyramid feature fusion architecture, and the processing process thereof comprises: enlarging the size of a deep high semantic feature map output by the backbone network through an upsampling operation; fusing the high layer feature after the upsampling, local spatial features output by the convolutional neural network path in the backbone network, and global timing context information output by the state space model path in the backbone network through a feature splicing operation to construct a feature pyramid containing rich details, semantics and timing logic; finally outputting the category information and position information of one or more targets in the image frame through the head network; based on the target position information output by the target detection network, extracting a region image sequence containing targets from the image frame sequence; inputting the sequence of region images into a MILA-SF behavior recognition network, the MILA-SF behavior recognition network comprising a slow path and a fast path, the slow path processing the input sequence of region images in a low frame rate and high channel number manner to capture spatial semantic information, the fast path processing in a high frame rate and low channel number manner to capture temporal dynamic information, and finally outputting the target's working behavior recognition result through feature fusion between the slow path and the fast path.

2. The method of claim 1, wherein, The processing process of the Mamba module includes: splitting the input sequence into an SSM branch input and a gating branch input through linear projection; performing depth convolution and SiLU activation function processing on the SSM branch input in turn, and then inputting into a bidirectional selective scanning structure, which transfers states based on dynamically generated parameters to capture context information; element-wise gating multiplication of the output of the bidirectional selective scanning structure and the gating branch input after activation function processing; and finally outputting the sequence after temporal information modeling and feature modulation through a projection layer.

3. The method of claim 1, wherein, The method further includes: The target detection network is trained using a Focaler-IoU loss function, and the calculation process of the Focaler-IoU loss function includes: setting a lower threshold d and an upper threshold u, where 0≤d<u≤1; The original intersection over union IoU value of the detection frame and the true value frame is reconstructed into a new intersection over union value using a linear interval mapping method , and the calculation formula is: In order to filter the extremely difficult samples with intersection over union value less than d and the simple samples with intersection over union value greater than u; Based on the new intersection over union value , calculate the final Focaler-IoU loss , the calculation formula is: .

4. The method of claim 1, wherein, The method further includes: preprocessing the video data: unifying the video stream into an mp4 format with a resolution of 1080 by 720 and a frame rate of 30fps; sampling and outputting a static image frame from the video at a time interval of every 2 seconds to form the sequence of image frames; using the LabelImg tool to perform target labeling on each static image frame to generate a labeling file in the YOLO standard format, which includes the class index of the target, the normalized center point x coordinate, the center point y coordinate, the width, and the height.

5. The method of claim 1, wherein, In the MILA-SF behavior recognition network, feature fusion is performed through a lateral connection set between the slow path and the fast path, and the lateral connection specifically includes: performing time dimension downsampling and channel dimension upsampling transformation on the features output by the fast path to match the feature dimension of the slow path; and fusing the transformed fast path features with the features of the slow path.

6. An electronic device, comprising: A processor, a memory, and a program or instructions stored on the memory and executable on the processor, when executed by the processor, implement the steps of the railway scene target detection and behavior recognition method based on spatiotemporal dual-flow features according to any one of claims 1-5.

7. A readable storage medium, characterized by, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the railway scene target detection and behavior recognition method based on spatiotemporal dual-flow features according to any one of claims 1-5.

Images