Railway personnel intrusion detection method and system based on visual perception and scene structure understanding

Abstract

The invention belongs to the technical field of visual detection, and particularly relates to a railway personnel intrusion detection method and system based on visual perception and scene structure understanding. The method comprises the following steps: carrying out decoding frame extraction, denoising and other operations on an input video stream; carrying out initial detection by adopting a self-developed YOLOv8 personnel detection model; extracting hidden points by using a multi-task convolutional network, and segmenting a steel rail boundary so as to obtain a rail limit area; dynamically adjusting the confidence coefficient of the detection frame based on the target track consistency and the scene structure information; modeling a target motion type and probability by using LSTM; and judging whether to trigger an intrusion alarm according to the information. The method solves the problems that the ability of tracking the long-distance sports personnel is insufficient, the confidence cannot be effectively adjusted, a single confidence threshold value cannot be set, missed detection is easily caused, and the overall detection effect is affected.

Description

Technical Field

[0001] This invention belongs to the technical field of visual inspection, and more specifically, relates to a railway personnel intrusion detection method and system based on visual perception and scene structure understanding. Background Technology

[0002] As a critical national infrastructure, railways face severe security challenges. Statistics show that 60% of railway accidents are caused by human negligence, and traditional manual inspections are limited by environmental visibility, manpower coverage, and response delays, making it difficult to achieve accurate, 24 / 7 monitoring. With the rapid development of railway transportation, railway line security monitoring has become particularly important. Nighttime intrusions into railways not only threaten the safety of railway transportation but may also cause serious harm to the intruders themselves.

[0003] Chinese patent document CN118429865A discloses a machine vision-based method for assessing and warning of foreign object intrusion risks on railways. The method includes: acquiring onboard monitoring video data; extracting track line features using machine vision methods; and fitting and expanding the trajectory points using Bessel functions and quadratic polynomials to obtain the complete railway track within the line of sight; acquiring monitoring video data along the railway line; obtaining a target detection algorithm using a pre-trained model fine-tuning method to detect the type and location information of foreign objects; and combining the railway track line detection results and the foreign object target detection results to construct a foreign object intrusion risk assessment system from both the foreign object type and location aspects, generating risk warning information based on risk matrix theory.

[0004] Existing railway intrusion detection systems suffer from high false positive and false negative rates in nighttime environments, especially at long distances. Factors such as small target size and poor lighting conditions lead to low detection confidence, resulting in missed detections. Current technologies primarily rely on single visual detection methods, lacking a deep understanding of scene structure and target behavior, making it difficult to accurately determine the target's intrusion intent and behavioral patterns. Furthermore, existing systems have insufficient tracking capabilities for long-distance moving individuals, failing to effectively adjust confidence levels. Setting a single confidence threshold easily leads to false positives and false negatives, thus affecting overall detection performance. Summary of the Invention

[0005] This invention aims to overcome at least one of the defects of the prior art and provide a railway personnel intrusion detection method based on visual perception and scene structure understanding. Through dynamic confidence adjustment, spatiotemporal consistency behavior pattern learning, false alarm suppression mechanism and scene understanding module, it improves the accuracy and reliability of railway personnel intrusion detection at night, reduces the false alarm rate, and enhances the adaptability to complex scenes. It solves the problems of insufficient tracking ability for long-distance moving personnel, inability to effectively adjust their confidence, and the easy occurrence of false negatives and missed detections when setting a single confidence threshold, which affects the overall detection effect.

[0006] The detailed technical solution of this invention is as follows: A railway personnel intrusion detection method based on visual perception and scene structure understanding, the method comprising: S1. Obtain real-time video stream from surveillance camera, decode it into single-frame image, and perform noise reduction processing on each frame image as input image; S2. Use the Yolov8L detection model to perform low confidence threshold detection on the input image, identify the intrusion target in the image, and output the target bounding box coordinates and initial confidence of the current frame. S3. Use a multi-task convolutional network to detect the hidden points and segmentation mask of the rail region in the image, then perform post-processing to obtain the detection results of the boundary region, and then adjust the confidence of the detection results through dynamic confidence. S4. Use an LSTM network to model the motion trend of the target and output the behavior probability distribution; S5. Analyze the confidence level, behavioral probability distribution, and danger zones of the target detection results to determine whether to trigger an intrusion alarm.

[0007] Furthermore, the multi-task convolutional network includes: a shared basic feature extraction layer, a regression layer, a boundary perceptron, and a decoder; The basic feature extraction layer and the regression layer constitute the hidden point detection subnetwork. The basic feature extraction layer extracts features from the input image, and the regression layer predicts the coordinates of the hidden points, as follows: (1); In formula (1), These are the coordinates of the hidden point. It is an input image of a railway scene.

[0008] Furthermore, in the hidden point detection subnetwork, the Smooth L1 loss is used to measure the difference between the predicted hidden point location and the true location. The Smooth L1 loss is a smooth L1 loss that uses a squared term when the error is small and a linear term when the error is large. This maintains sensitivity to small errors while avoiding excessive influence of large errors on the gradient, and is expressed as: (2); In formula (2), These are the predicted coordinates of the hidden surface. These are the actual coordinates of the hidden surface point.

[0009] The basic feature extraction layer, boundary perceptron, and decoder constitute the rail segmentation sub-network. The convolutional layer extracts features from the input image, and the boundary perceptron detects boundary features. The boundary features detected by the boundary perceptron are fused with the extracted features, and the output is a segmentation mask for the rail region, represented as: (3); In formula (3), It is a segmentation mask for the rail area. It is a rail segmentation network.

[0010] Furthermore, the basic feature extraction layer, boundary perceptron, and decoder constitute the rail segmentation sub-network, specifically including: The boundary perceptron is composed of multiple Canny operators connected in series, and the decoder is composed of multiple decoder layers and upsampling layers connected in series. The input to the first decoder is the features extracted from the basic feature extraction layer, the input to the second decoder is the output of the first Canny operator, and the input to the third decoder is the output of the second Canny operator, and they are interconnected in sequence. The inputs to the first Canny operator are the features extracted by the basic feature extraction layer and the output features of the upsampling layer in the first decoder; the inputs to the second Canny operator are the features extracted by the basic feature extraction layer and the output features of the upsampling layer in the second decoder, and these are sequentially interconnected. Finally, the decoder obtains the segmentation mask for the rail area.

[0011] Furthermore, the Canny operator uses the Canny edge detection algorithm to extract features of the rail boundary. The Canny algorithm detects edge pixels by calculating the gradient magnitude and direction of image features. The feature map obtained after the basic feature extraction layer is subjected to Canny edge detection to generate an edge feature map, which is a binary map with edge pixel values ​​of 1 and non-edge pixel values ​​of 0. The edge feature map is weighted and summed with the output feature map of the decoder upsampling layer to achieve weighted enhancement of the features. Then, the image is restored to its original scale layer by layer through the upsampling layer in the decoder. By fusing edge features and high-level semantic features, this invention further improves the accuracy and boundary fit of rail region segmentation.

[0012] Furthermore, for the loss function of the rail segmentation task, Dice loss is used, and the segmentation effect is measured by calculating the similarity between the predicted segmentation mask and the real segmentation mask; Meanwhile, Focal loss and boundary-aware loss are introduced to optimize the segmentation effect on the rail boundary region. Focal loss focuses on the segmentation of pixels in the rail region, reduces the weight of the background class, and improves the model's ability to recognize pixels in the rail region. Boundary-aware loss is achieved by calculating the difference between the predicted segmentation mask and the real segmentation mask in the boundary region. Dice loss is expressed as: (4); In formula (4), It is the predicted segmentation mask. It is the actual labeled segmentation mask. It is a smoothing term; Focal Loss is represented as: (5); In formula (5), It balances the weights of positive and negative samples. It is a focusing parameter that adjusts the difficulty level of samples; The boundary-aware loss is represented as: (6); In formula (6), It is boundary-aware loss. It is the first Edge feature values ​​of 1 pixel, It is the segmentation mask predicted from i pixels. It is the actual segmentation mask. It is the total number of pixels; Combining Dice loss, Focal Loss, and edge-aware loss, we form the total loss function for rail region segmentation, expressed as: (7); In formula (7), , , These are the weights of each loss function, used to balance the weights of different losses.

[0013] Furthermore, the overall loss function of the multi-task convolutional network can be expressed as: (8); To further optimize multi-task learning, dynamically adjustable task weights are added. These weights are dynamically adjusted based on the loss value of each task, thereby automatically balancing the contributions of the two tasks during training.

[0014] (9); This dynamic adjustment method ensures that both tasks receive appropriate attention during training, thereby improving the effectiveness of multi-task learning.

[0015] Furthermore, the post-processing to obtain the detection results of the boundary region specifically includes: ① Extract the boundary contour point sets of the left and right rails from the segmentation mask: Use cv2.findContours to extract the contours, and distinguish the left and right rails according to the x coordinate values ​​of the contours to obtain two point sets, the left rail point set and the right rail point set; ② Fit the curves of the left and right rails using a polynomial fitting method to obtain the curves of the left and right rails; ③ Construct the track centerline: For multiple y-coordinate values ​​in the image, calculate the x-coordinates of the corresponding positions on the left and right rails, and calculate the midpoint. Connect the midpoints to obtain the center line of the track; ④ Generate boundary along the centerline: Calculate the tangent slope at each point on the centerline, calculate the normal direction based on the slope, and offset a certain distance in the normal direction to obtain two new points, which belong to the left and right boundary lines respectively. Traverse all centerline points and collect boundary point data; preferably, the offset distance is obtained by using an offset distance function that varies with y. ⑤ Combine the hidden point to obtain the bounded region: Extend the bounded boundary line to the hidden point, and combine the left and right bounded boundary points to form a closed trapezoidal or polygonal region.

[0016] Furthermore, the dynamic adjustment of the confidence level of the detection results specifically includes: sequentially performing Kalman filter prediction and confidence level adjustment, pixel temporal change information adjustment, and scene structure information fusion adjustment.

[0017] Kalman filter prediction and confidence adjustment: The Kalman filter predicts the position of a device in the next frame based on its historical position and motion state; Based on the prediction results of the Kalman filter and the actual detection results, the confidence level of the target is dynamically adjusted as follows: (10); In formula (10), The target location detected by the Yolov8L detection model. The target location predicted by the Kalman filter. For confidence level adjustment, This is the position deviation threshold.

[0018] Pixel temporal variation information adjustment: By analyzing the pixel changes of the target over time, a dynamic confidence adjustment factor is constructed to further adjust the confidence of the detection results. Preferably, the pixel temporal change information adjustment is achieved by analyzing the pixel changes of the target in consecutive frames, calculating the average rate of change of the distance over the analysis time period N, and determining whether the target's motion is smooth and continuous, as follows: Pixel timing changes (11); In formula (11), It is the Euclidean distance between the target at time step t and time step t+1, and N is the length of the time series being analyzed.

[0019] The "pixel temporal variation" value quantifies the degree of fluctuation in the target's motion speed. The smaller the "pixel temporal variation" value, the smaller the displacement change of the target between consecutive frames, and the smoother and more continuous the motion trajectory, which conforms to the physical motion law of the real target. Conversely, if the value is large, it indicates that the target's motion speed has a violent and irregular jump, and the trajectory is not smooth and discontinuous, which is likely caused by noise, false detection or tracking instability.

[0020] Based on the above evaluation, a dynamic confidence adjustment factor related to pixel temporal changes is constructed. This factor follows the following principle: when the trajectory is smooth and continuous, the adjustment factor is ≥1 to enhance or maintain the detection confidence; when the trajectory is not smooth or discontinuous, the adjustment factor is <1 to weaken the detection confidence. Multiply the original detection confidence level by the dynamic confidence level adjustment factor to obtain the final adjusted confidence level, as shown in formula (12): Confidence level = Confidence level · Dynamic confidence level adjustment factor (12).

[0021] Scene structure information fusion adjustment: The confidence weight of targets falling into the boundary danger zone is automatically increased using scene structure information. Specifically: Preferably, the scene structure information fusion specifically includes: Scene structure information weight (13); In formula (13), In order to be in the danger zone, To be outside the danger zone, It is the weight coefficient of the dangerous area. A value greater than 1 indicates that the target confidence in the dangerous area is higher. The dangerous area refers to the boundary area jointly generated by the hidden point obtained from the understanding of the multi-task scenario and the rail segmentation result.

[0022] Furthermore, S4 specifically includes: A lightweight LSTM network is constructed to model the typical motion patterns of targets in video sequences: low-level behavioral features such as target bounding box coordinates, velocity, and acceleration; and high-level behavioral patterns such as walking, vaulting, crouching, and lingering.

[0023] (1) Using low-level behavioral features as input, a hierarchical LSTM structure is introduced to extract and fuse high-level behavioral patterns layer by layer, and finally output the target behavior probability distribution; low-level behavioral features include bounding box coordinates, velocity, and acceleration, and high-level behavioral patterns include walking, climbing, crouching, and lingering. (2) Combine the behavioral probability distribution obtained in (1) with the target's true size and motion trend estimated by the spatial scale mapping model to correct the behavioral probability distribution; (3) Using the behavior probability distribution modified in (2) as the weight, the final behavior probability distribution is generated by weighted fusion with the visual detection confidence.

[0024] Lightweight LSTM networks are used to model the motion patterns of targets in video sequences. When low-level and high-level features are input into the LSTM, the improved LSTM network structure includes an input layer, a feature fusion layer, LSTM units, and an output layer, as follows: Input layer: Low-level feature vectors are The high-level feature vector is For each time step t, the low-level features and high-level features are concatenated into a new feature vector: (14); Then this concatenated feature vector As input to the LSTM input layer.

[0025] Feature fusion layer: The improved feature fusion process is as follows: Attention-weighted fusion introduces an attention mechanism to weight and fuse low-level and high-level features, enabling LSTM to automatically learn the importance of different features. Attention weight coefficients are calculated as follows: (15); In formula (15), It is the attention weight matrix. It is the attention bias vector. Used to normalize attention weights. Feature weighted fusion calculation: (16); In formula (16), It is element-wise multiplication.

[0026] Output layer: Outputs the probability distribution of target behavior, such as walking, crouching, vaulting, etc.

[0027] LSTM units include input gates Forgotten Gate Candidate cell status Cell state Output gate and hidden state Among them, the hidden state Indicates the output of the LSTM unit; Traditional LSTM directly inputs the raw features into the network, while the improved LSTM uses an attention mechanism to weight different features, enabling the model to pay more attention to important features. Through feature concatenation and fusion, it can process multiple features, allowing the model to comprehensively utilize information at different levels and capture the target's behavior patterns more comprehensively, making it particularly suitable for behavior recognition in complex scenarios.

[0028] The output layer of an LSTM network uses the Softmax function to store the hidden states. Convert to behavioral probability distribution: (17); In formula (17), The target performs actions from time 1 to time t. The probability, It is the total number of behavior categories. and These are the weights and biases of the output layer, for each time step. The LSTM outputs the probability distribution of the target performing different actions at that moment, such as walking, crouching, and climbing.

[0029] Compared to traditional LSTMs that only output a single behavior category or simple feature representation, the behavior pattern modeling module can output detailed behavior probability distributions, providing richer information for subsequent alarm decisions and facilitating detailed analysis and judgment of different behavior patterns.

[0030] Furthermore, S5 specifically includes: By comprehensively analyzing the confidence level of the target detection results, the probability distribution of behavior, and the judgment of dangerous areas, it is determined whether to trigger an intrusion alarm; Confidence threshold judgment: Set a confidence threshold, usually 0.8. If the confidence threshold is greater than this, return True; Behavioral pattern judgment: Set a confidence threshold for behavioral patterns, usually 0.7. If the highest probability behavior in the target's behavioral probability distribution is greater than this threshold, return True. Danger Zone Judgment: Check if the target is located within a danger zone. If the target area and the danger zone intersect, return True. Comprehensive decision: The comprehensive decision result is True and an intrusion alarm is triggered only when the confidence level judgment, behavior pattern judgment, and danger zone judgment are all True.

[0031] In another aspect of the present invention, a system for railway personnel intrusion detection based on visual perception and scene structure understanding is provided, the system comprising: a data acquisition module, a scene understanding module, a dynamic confidence adjustment module, a behavior pattern modeling module, and a comprehensive decision-making module; The data acquisition module is used to acquire image data in real time; the scene understanding module includes a Yolov8L detection model and a multi-task convolutional network, which are used to identify intrusion targets and acquire boundary regions, respectively; the dynamic confidence adjustment module is used to adjust the confidence of the detection results; the behavior pattern modeling module is used to detect the probability distribution of behavior; and the comprehensive decision module is used to determine whether to trigger an intrusion alarm and issue an alarm.

[0032] In another aspect of the invention, a computer-readable storage medium is also provided, which stores executable instructions that, when executed, cause the machine to perform a railway personnel intrusion detection method based on visual perception and scene structure understanding as described above.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention provides a railway personnel intrusion detection method and system based on visual perception and scene structure understanding. By introducing scene understanding and dynamic confidence adjustment, the accuracy of railway personnel intrusion detection at night is significantly improved, especially the detection accuracy in night or low light scene, and the recognition ability of small and slow targets is enhanced. At the same time, through dynamic confidence adjustment and behavior pattern modeling, the false alarm rate is effectively reduced, the anti-interference ability in complex scene is enhanced, and the ability to adapt to complex scene is enhanced.

[0034] (2) The present invention provides a railway personnel intrusion detection method and system based on visual perception and scene structure understanding. In the rail segmentation task, a boundary perceptron is added, and the features of the rail boundary are extracted by Canny edge detection. The edge features are fused with the feature map of the segmentation network to enhance the network's perception of the rail boundary. In the loss function, Dice loss, Focal Loss and boundary perception loss are combined to further optimize the segmentation results and improve the recognition ability of pixels in the track area, especially to deal with the problem of class imbalance.

[0035] (3) The railway personnel intrusion detection method and system based on visual perception and scene structure understanding provided by the present invention uses a dynamic confidence adjustment module to predict the target position through a Kalman filter and adjusts the confidence based on the consistency between the predicted position and the actual detection position. This dynamic adjustment mechanism can effectively cope with the movement changes of the target and improve the detection accuracy. The behavior pattern modeling module models the movement behavior and probability of the target through an LSTM network. It introduces a hierarchical LSTM structure and forms behavior modeling through low-level behavior features and high-level behavior patterns. Hierarchical modeling can capture the details and long-term dependencies of the target behavior more finely. When the target behavior includes multiple stages, it can more accurately judge its intrusion intention and improve the recognition accuracy of complex behavior patterns. In addition, by combining behavior probability and visual detection confidence, this fusion mechanism can effectively distinguish between real intruders and non-intrusion targets and reduce the false alarm rate. Attached Figure Description

[0036] Figure 1 This is a flowchart of the railway personnel intrusion detection method based on visual perception and scene structure understanding as described in this invention.

[0037] Figure 2 This is a schematic diagram of the multi-task network in the scene structure extraction module of Embodiment 1 of the present invention.

[0038] Figure 3 This is a schematic diagram of the boundary region generated based on the results (coordinates of the hidden point and the rail boundary region) obtained from the multi-task network in Embodiment 1 of the present invention.

[0039] Figure 4 This is a comparison chart of the detection effect of the Yolov8L detection model in Embodiment 1 of the present invention and the detection effect of the present invention.

[0040] Figure 5 This is a comparison diagram of the close-range detection results between the Yolov8L detection model in Embodiment 1 of this invention and the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0043] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0044] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0045] Embodiment 1 Refer Figure 1 , this embodiment provides a railway personnel intrusion detection method based on visual perception and scene structure understanding, and the method includes: S1. Video stream processing: Obtain a real-time video stream from a monitoring camera, decode it into single-frame images, and perform denoising processing on each frame image as the input image.

[0046] S2. Target detection: Use the Yolov8L detection model to detect the processed image with a low confidence threshold, identify the intrusion targets in the picture, and output the target bounding box coordinates and initial confidence of the current frame; S3. Scene structure extraction: Use a multi-task convolutional network to detect the vanishing point of the image and the segmentation mask of the rail area, then perform post-processing to obtain the detection result of the limited area, that is, the generation of the limited area, and then adjust the confidence of the detection result through dynamic confidence adjustment.

[0047] The vanishing point is the intersection point of a straight line in the three-dimensional world on the image plane, which contains the perspective geometric information of the scene. In the railway scene, the position of the vanishing point is crucial for understanding the scene structure and restoring spatial information. By detecting the vanishing point, the extension direction of the railway track and the depth information of the scene can be inferred, thus providing important geometric clues for the detection and analysis of intrusion targets.

[0048] The goal of the rail area segmentation task is to accurately segment the rail area in the railway scene image. The rail position is the main basis for dividing the intrusion alarm area. Accurate rail segmentation can effectively identify the dangerous area within the railway limit, thereby improving the accuracy and reliability of intrusion detection.

[0049] Preferably, use a multi-task convolutional network to detect the vanishing point of the image and the segmentation mask of the rail boundary area to obtain the limited area, as Figure 2As shown, the multi-task convolutional network includes: a shared basic feature extraction layer, a regression layer, a boundary perceptron, and a decoder. Multi-task learning reduces the consumption of computing resources through the shared basic feature extraction layer, and improves the generalization ability and detection accuracy of the model by jointly training multiple related tasks. The basic feature extraction layer and the regression layer constitute a hidden point detection subnetwork. The basic feature extraction layer extracts features from the input image, and the regression layer predicts the coordinates of the hidden points. Preferably, the basic feature extraction layer consists of multiple convolutional layers, as shown below: (1); In formula (1), These are the coordinates of the hidden point. It is an input image of a railway scene.

[0050] Furthermore, in the hidden point detection subnetwork, the Smooth L1 loss is used to measure the difference between the predicted hidden point location and the true location. The Smooth L1 loss is a smooth L1 loss that uses a squared term when the error is small and a linear term when the error is large. This maintains sensitivity to small errors while avoiding excessive influence of large errors on the gradient, and is expressed as: (2); In formula (2), These are the predicted coordinates of the hidden surface. These are the actual coordinates of the hidden surface point.

[0051] The basic feature extraction layer, boundary perceptron, and decoder constitute the rail segmentation sub-network. The convolutional layer extracts features from the input image, and the boundary perceptron detects boundary features. The boundary features detected by the boundary perceptron are fused with the extracted features, and the output is a segmentation mask for the rail region, represented as: (3); In formula (3), It is a segmentation mask for the rail area. It is a rail segmentation network.

[0052] Furthermore, the basic feature extraction layer, boundary perceptron, and decoder constitute a rail segmentation sub-network. This invention adds a sub-branch, the boundary perceptron, to the original encoding-decoding network structure. This sub-branch uses the Canny edge detection algorithm to extract features of the rail boundaries, fuses the boundary features obtained from edge detection with the feature map of the segmentation network, enhances the network's ability to perceive rail boundaries, and adds a boundary perception loss to the loss function to further optimize the segmentation results. Specifically, this includes: The boundary perceptron is composed of multiple Canny operators connected in series, and the decoder is composed of multiple decoder layers and upsampling layers connected in series. The input to the first decoder is the features extracted from the basic feature extraction layer, the input to the second decoder is the output of the first Canny operator, and the input to the third decoder is the output of the second Canny operator, and they are interconnected in sequence. The inputs to the first Canny operator are the features extracted by the basic feature extraction layer and the output features of the upsampling layer in the first decoder; the inputs to the second Canny operator are the features extracted by the basic feature extraction layer and the output features of the upsampling layer in the second decoder, and these are sequentially interconnected. Finally, the decoder obtains the segmentation mask for the rail area.

[0053] Furthermore, the Canny operator uses the Canny edge detection algorithm to extract features of the rail boundary. The Canny algorithm detects edge pixels by calculating the gradient magnitude and direction of image features. The feature map obtained after the basic feature extraction layer is subjected to Canny edge detection to generate an edge feature map, which is a binary map with edge pixel values ​​of 1 and non-edge pixel values ​​of 0. The edge feature map is weighted and summed with the output feature map of the decoder upsampling layer to achieve weighted enhancement of the features. Then, the image is restored to its original scale layer by layer through the upsampling layer in the decoder. By fusing edge features and high-level semantic features, this invention further improves the accuracy and boundary fit of rail region segmentation.

[0054] Furthermore, for the loss function of the rail segmentation task, Dice loss is used, and the segmentation effect is measured by calculating the similarity between the predicted segmentation mask and the real segmentation mask; Meanwhile, Focal loss and boundary-aware loss are introduced to optimize the segmentation effect on the rail boundary region. Focal loss focuses on the segmentation of pixels in the rail region, reduces the weight of the background class, and improves the model's ability to recognize pixels in the rail region. Boundary-aware loss is achieved by calculating the difference between the predicted segmentation mask and the real segmentation mask in the boundary region, thereby optimizing the segmentation effect on the rail boundary region. Dice loss is expressed as: (4); In formula (4), It is the predicted segmentation mask. It is the actual labeled segmentation mask. It is a smoothing term; Focal Loss can be represented as: (5); In formula (5), It balances the weights of positive and negative samples. It is a focusing parameter that adjusts the difficulty level of samples; The boundary-aware loss is represented as: (6); In formula (6), It is boundary-aware loss. It is the first Edge feature values ​​of 1 pixel, It is the segmentation mask predicted from i pixels. It is the actual segmentation mask. It is the total number of pixels; Combining Dice loss, Focal Loss, and edge-aware loss, we form the total loss function for rail region segmentation, expressed as: (7); In formula (7), , , These are the weights of each loss function, used to balance the weights of different losses.

[0055] Furthermore, the overall loss function of the multi-task convolutional network can be expressed as: (8); To further optimize multi-task learning, dynamically adjustable task weights are added. These weights are dynamically adjusted based on the loss value of each task, thereby automatically balancing the contributions of the two tasks during training.

[0056] (9); This dynamic adjustment method ensures that both tasks receive appropriate attention during training, thereby improving the effectiveness of multi-task learning.

[0057] Furthermore, the post-processing to obtain the detection results of the boundary region specifically includes: ① Extract the boundary contour point sets of the left and right rails from the segmentation mask: Use cv2.findContours to extract the contours, and distinguish the left and right rails according to the x coordinate values ​​of the contours to obtain two point sets, the left rail point set and the right rail point set; ② Fit the curves of the left and right rails using a polynomial fitting method to obtain the curves of the left and right rails; ③ Construct the track centerline: For multiple y-coordinate values ​​in the image, calculate the x-coordinates of the corresponding positions on the left and right rails, and calculate the midpoint. Connect the midpoints to obtain the center line of the track; ④ Generate boundary along the centerline: Calculate the tangent slope at each point on the centerline, calculate the normal direction based on the slope, and offset a certain distance in the normal direction to obtain two new points, belonging to the left and right boundary lines respectively. Traverse all centerline points and collect boundary point data. Preferably, the offset distance is obtained through an offset distance function that varies with y, and should vary according to the depth, becoming narrower as the distance increases. ⑤ Obtain the bounded region by combining the hidden point: Extend the bounded boundary line to the hidden point, and combine the left and right bounded boundary points to form a closed trapezoidal or polygonal region, such as... Figure 3 The purple area is shown.

[0058] Furthermore, the dynamic adjustment of the confidence level of the detection results specifically includes: sequentially performing Kalman filter prediction and confidence level adjustment, pixel temporal change information adjustment, and scene structure information fusion adjustment.

[0059] Kalman filter prediction and confidence adjustment: The Kalman filter predicts the position of a device in the next frame based on its historical position and motion state; Based on the prediction results of the Kalman filter and the actual detection results, the confidence level of the target is dynamically adjusted as follows: (10); In formula (10), The target location detected by the Yolov8L detection model. The target location predicted by the Kalman filter. For confidence level adjustment, This is the position deviation threshold.

[0060] Pixel temporal variation information adjustment: By analyzing the pixel changes of the target over time, a dynamic confidence adjustment factor is constructed to further adjust the confidence of the detection results. Preferably, the pixel temporal change information adjustment determines whether the target's motion is smooth and continuous by analyzing the pixel changes of the target in consecutive frames, as follows: Pixel timing changes (11); In formula (11), It is the Euclidean distance between the target at time step t and time step t+1, and N is the length of the time series being analyzed. The average rate of change of this distance over the analysis period N is calculated.

[0061] The "pixel temporal variation" value quantifies the degree of fluctuation in the target's motion speed. The smaller the "pixel temporal variation" value, the smaller the displacement change of the target between consecutive frames, and the smoother and more continuous the motion trajectory, which conforms to the physical motion law of real targets (such as pedestrians). Conversely, if the value is large, it indicates that the target's motion speed has drastic and irregular jumps, and the trajectory is not smooth and discontinuous, which is likely caused by noise, false detection or tracking instability.

[0062] Constructing a dynamic confidence adjustment factor: Based on the above evaluation, a dynamic confidence adjustment factor related to "pixel temporal change" is constructed. This factor follows the following principles: when the trajectory is smooth and continuous (i.e., the value of "pixel temporal change" is small), the adjustment factor is ≥1, which enhances or maintains the detection confidence; when the trajectory is not smooth and discontinuous (i.e., the value of "pixel temporal change" is large), the adjustment factor is <1, which weakens the detection confidence. Multiplying the original detection confidence by this dynamic confidence adjustment factor, the final adjusted confidence is obtained, as shown in formula (12): Confidence level = Confidence level · Dynamic confidence level adjustment factor (12).

[0063] Scene structure information fusion adjustment: The confidence weight of targets falling into the boundary danger zone is automatically increased using scene structure information. Specifically: Preferably, the scene structure information fusion specifically includes: Scene structure information weight (13); In formula (13), In order to be in the danger zone, To be outside the danger zone, It is the weight coefficient of the dangerous area. A value greater than 1 indicates that the target confidence in the dangerous area is higher. The dangerous area refers to the boundary area jointly generated by the hidden point obtained from the understanding of the multi-task scenario and the rail segmentation result.

[0064] S4. Behavioral Pattern Modeling: Use an LSTM network to model the motion trend of the target, output the behavioral probability distribution, build a lightweight LSTM network, and model the typical motion patterns of the target in the video sequence.

[0065] Furthermore, S4 specifically includes: (1) Using low-level behavioral features as input, a hierarchical LSTM structure is introduced to extract and fuse high-level behavioral patterns layer by layer, and finally output the target behavior probability distribution; low-level behavioral features include bounding box coordinates, velocity, and acceleration, and high-level behavioral patterns include walking, climbing, crouching, and lingering. (2) Combine the behavioral probability distribution obtained in (1) with the target's true size and motion trend estimated by the spatial scale mapping model to correct the behavioral probability distribution; (3) Using the behavior probability distribution modified in (2) as the weight, it is weighted and fused with the visual detection confidence, i.e. the low confidence detection of the Yolov8L detection model, to generate the final behavior probability distribution.

[0066] Preferably, the introduced hierarchical LSTM structure is as follows: a lightweight LSTM network is used to model the motion patterns of the target in the video sequence. When low-level and high-level features are input into the LSTM, the improved LSTM network structure includes an input layer, a feature fusion layer, LSTM units, and an output layer, as follows: Input layer: Low-level feature vectors are The high-level feature vector is For each time step t, the low-level features and high-level features are concatenated into a new feature vector: (14); Then this concatenated feature vector As input to the LSTM input layer.

[0067] Feature fusion layer: The improved feature fusion process is as follows: Attention-weighted fusion introduces an attention mechanism to weight and fuse low-level and high-level features, enabling LSTM to automatically learn the importance of different features. Attention weight coefficients are calculated as follows: (15); In formula (15), It is the attention weight matrix. It is the attention bias vector. Used to normalize attention weights. Feature weighted fusion calculation: (16); In formula (16), It is element-wise multiplication.

[0068] Output layer: Outputs the probability distribution of target behavior, such as walking, crouching, vaulting, etc.

[0069] LSTM units include input gates Forgotten Gate Candidate cell status Cell state Output gate and hidden state Among them, the hidden state Indicates the output of the LSTM unit; Traditional LSTM directly inputs the raw features into the network, while the improved LSTM uses an attention mechanism to weight different features, enabling the model to pay more attention to important features. Through feature concatenation and fusion, it can process multiple features, allowing the model to comprehensively utilize information at different levels and capture the target's behavior patterns more comprehensively, making it particularly suitable for behavior recognition in complex scenarios.

[0070] The output layer of an LSTM network uses the Softmax function to store the hidden states. Convert to behavioral probability distribution: (17); In formula (17), The target performs actions from time 1 to time t. The probability, It is the total number of behavior categories. and These are the weights and biases of the output layer, for each time step. The LSTM outputs the probability distribution of the target performing different actions at that moment, such as walking, crouching, and climbing.

[0071] Compared to traditional LSTMs that only output a single behavior category or simple feature representation, the behavior pattern modeling module can output detailed behavior probability distributions, providing richer information for subsequent alarm decisions and facilitating detailed analysis and judgment of different behavior patterns.

[0072] S5. Alarm Decision: Based on a comprehensive analysis of the confidence level of the target's detection results, the probability distribution of behavior, and the danger zone, determine whether to trigger an intrusion alarm.

[0073] Confidence threshold judgment: Set a confidence threshold, usually 0.8. If the confidence threshold is greater than this, return True; Behavioral pattern judgment: Set a confidence threshold for behavioral patterns, usually 0.7. If the highest probability behavior in the target's behavioral probability distribution is greater than this threshold, return True. Danger Zone Judgment: Check if the target is located within a danger zone. If the target area and the danger zone intersect, return True. Comprehensive decision: The comprehensive decision result is True and an intrusion alarm is triggered only when the confidence level judgment, behavior pattern judgment, and danger zone judgment are all True.

[0074] The alarm decision module is a key component of the railway nighttime intrusion detection system. It is responsible for comprehensively analyzing the output information of various front-end modules and ultimately deciding whether to trigger an intrusion alarm. The decision logic of this module is based on the target's confidence level, behavioral patterns, and scene structure information, ensuring that an alarm is only triggered when there is a high degree of certainty that the target is a real intruder, thereby improving the reliability and accuracy of the system.

[0075] The inputs to the alarm decision module include: Output of the dynamic confidence adjustment module: target bounding box coordinates, adjusted confidence score, and target trajectory information; The output of the behavior pattern modeling module includes: the target's behavior probability distribution (such as walking, crouching, etc.) and a comprehensive probability score. The output of the scene structure feature extraction module includes: hidden point coordinates, rail region segmentation mask, and dangerous area definition.

[0076] Decision-making logic: 1. Confidence threshold judgment: Set a confidence threshold, usually 0.8. If the confidence threshold is greater than this threshold, return True; 2. Behavioral Pattern Judgment: Set a confidence threshold for behavioral patterns, usually 0.7. If the highest probability behavior in the target's behavioral probability distribution is greater than this threshold, return True. 3. Danger Zone Judgment: Check if the target is located within a danger zone. If the target area and the danger zone intersect, return True. 4. Comprehensive Decision: The comprehensive decision result will be True and trigger an intrusion alarm only when the confidence level judgment, behavior pattern judgment, and danger area judgment are all True.

[0077] The output of the alarm decision module is the alarm status and alarm information, which includes target location information, behavior patterns, and comprehensive confidence score.

[0078] Specifically, the comparison diagram between the detection results of the original model (i.e., the model without improvement) and the results processed by the method of this invention is shown in the figure below. Figure 4 , Figure 5 As shown, Figure 4 For results of long-distance detection, Figure 5 This is a close-range detection result; Figure 4 and Figure 5 The left image shows the detection results of the original model. It can be seen that the original model could not detect people walking on the left track and mistakenly detected the reflective signs on the catenary poles as people. However, after optimization by the present invention, the dynamic confidence adjustment module can recall people moving on the left side of the track. The output of the module that models the human behavior pattern can suppress false alarms of people in the stationary state of the catenary poles.

[0079] Example 2 This embodiment provides a system for implementing a railway personnel intrusion detection method based on visual perception and scene structure understanding. The system includes: a data acquisition module, a scene understanding module, a dynamic confidence adjustment module, a behavior pattern modeling module, and a comprehensive decision-making module. The data acquisition module is used to acquire image data in real time; the scene understanding module includes a Yolov8L detection model and a multi-task convolutional network, which are used to identify intrusion targets and acquire boundary regions, respectively; the dynamic confidence adjustment module is used to adjust the confidence of the detection results; the behavior pattern modeling module is used to detect the probability distribution of behavior; and the comprehensive decision module is used to determine whether to trigger an intrusion alarm and issue an alarm.

[0080] Example 3 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, cause the machine to perform the railway personnel intrusion detection method based on visual perception and scene structure understanding as described above.

[0081] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0082] In this case, the program code itself, which can be read from the readable medium, can perform the functions of any of the above embodiments, and therefore the computer-readable code and the readable storage medium storing the computer-readable code constitute a part of this specification.

[0083] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A railway personnel intrusion detection method based on visual perception and scene structure understanding, characterized in that, The method includes: S1. Obtain real-time video stream from surveillance camera, decode it into single-frame image, and perform noise reduction processing on each frame image as input image; S2. Use the Yolov8L detection model to perform low confidence threshold detection on the input image, identify the intrusion target in the image, and output the target bounding box coordinates and initial confidence of the current frame. S3. Use a multi-task convolutional network to detect the hidden points and segmentation mask of the rail region in the image, then perform post-processing to obtain the detection results of the boundary region, and then adjust the confidence of the detection results through dynamic confidence. S4. Use an LSTM network to model the motion trend of the target and output the behavior probability distribution; S5. Analyze the confidence level, behavioral probability distribution, and danger zones of the target detection results to determine whether to trigger an intrusion alarm.

2. The railway personnel intrusion detection method based on visual perception and scene structure understanding according to claim 1, characterized in that, The multi-task convolutional network includes: a shared basic feature extraction layer, a regression layer, a boundary perceptron, and a decoder; The basic feature extraction layer and the regression layer constitute the hidden point detection subnetwork. The basic feature extraction layer extracts features from the input image, and the regression layer predicts the coordinates of the hidden points, as follows: （1）； In formula (1), These are the coordinates of the hidden point. It is an input image of a railway scene; In the hidden point detection subnetwork, the Smooth L1 loss is used to measure the difference between the predicted hidden point location and the true location, denoted as: : （2）； In formula (2), These are the predicted coordinates of the hidden surface. These are the actual coordinates of the hidden surface points; The basic feature extraction layer, boundary perceptron, and decoder constitute the rail segmentation sub-network. The convolutional layer extracts features from the input image, and the boundary perceptron detects boundary features. The boundary features detected by the boundary perceptron are fused with the extracted features, and the output is a segmentation mask for the rail region, represented as: （3）； In formula (3), It is a segmentation mask for the rail area. It is a rail segmentation network.

3. The railway personnel intrusion detection method based on visual perception and scene structure understanding according to claim 2, characterized in that, The boundary perceptron is composed of multiple Canny operators connected in series, and the decoder is composed of multiple decoder layers and upsampling layers connected in series. The input to the first decoder is the features extracted from the basic feature extraction layer, the input to the second decoder is the output of the first Canny operator, and the input to the third decoder is the output of the second Canny operator, and they are interconnected in sequence. The inputs to the first Canny operator are the features extracted by the basic feature extraction layer and the output features of the upsampling layer in the first decoder; the inputs to the second Canny operator are the features extracted by the basic feature extraction layer and the output features of the upsampling layer in the second decoder, and these are sequentially interconnected; finally, the segmentation mask of the rail region is obtained by the last decoder. The Canny operator uses the Canny edge detection algorithm to extract the features of the rail boundary. The Canny algorithm detects edge pixels by calculating the gradient magnitude and direction of the image features. The feature map obtained after the basic feature extraction layer is subjected to Canny edge detection to generate an edge feature map, which is a binary map with edge pixel values ​​of 1 and non-edge pixel values ​​of 0. The edge feature map is weighted and summed with the output feature map of the decoder's upsampling layer, and then restored to the original image scale layer by layer through the upsampling layer in the decoder.

4. The railway personnel intrusion detection method based on visual perception and scene structure understanding according to claim 3, characterized in that, For the loss function of the rail segmentation subnetwork, Dice loss is used, and the segmentation effect is measured by calculating the similarity between the predicted segmentation mask and the real segmentation mask. Meanwhile, Focal loss and boundary-aware loss are introduced to optimize the segmentation effect on the rail boundary region; Dice loss is expressed as: （4）； In formula (4), It is the predicted segmentation mask. It is the actual labeled segmentation mask. It is a smoothing term; Focal Loss is represented as: （5）； In formula (5), It balances the weights of positive and negative samples. It is a focusing parameter that adjusts the difficulty level of samples; The boundary-aware loss is represented as: （6）； In formula (6), It is boundary-aware loss. It is the first Edge feature values ​​of 1 pixel, It is the segmentation mask predicted from i pixels. It is the actual segmentation mask. It is the total number of pixels; By combining Dice loss, Focal Loss, and edge-aware loss, a total loss function for rail region segmentation is formed. , is represented as: （7）； In formula (7), , , These are the weights of each loss function, used to balance the weights of different losses; Specifically, the overall loss function of a multi-task convolutional network is expressed as: （8）； The task weights are dynamically adjusted based on the loss value of each task: （9）。 5. The railway personnel intrusion detection method based on visual perception and scene structure understanding according to claim 4, characterized in that, The post-processing to obtain the detection results of the boundary region specifically includes: ① Extract the boundary contour point sets of the left and right rails from the segmentation mask: Use cv2.findContours to extract the contours, and distinguish the left and right rails according to the x coordinate values ​​of the contours to obtain two point sets, the left rail point set and the right rail point set; ② Fit the curves of the left and right rails using a polynomial fitting method to obtain the curves of the left and right rails; ③ Construct the track centerline: For multiple y-coordinate values ​​in the image, calculate the x-coordinates of the corresponding positions on the left and right rails, and calculate the midpoint. Connect the midpoints to obtain the center line of the track; ④ Generate boundary along the centerline: Calculate the tangent slope of each point on the centerline, calculate the normal direction based on the slope, offset a certain distance in the positive and negative directions of the normal direction to obtain two new points, which belong to the left and right boundary respectively. Traverse all centerline points and collect boundary points. ⑤ Combine the hidden point to obtain the bounded region: Extend the bounded boundary line to the hidden point, and combine the left and right bounded boundary points to form a closed trapezoidal or polygonal region.

6. The railway personnel intrusion detection method based on visual perception and scene structure understanding according to claim 5, characterized in that, The specific steps of dynamically adjusting the confidence level of the detection results include: sequentially performing Kalman filter prediction and confidence level adjustment, pixel temporal change information adjustment, and scene structure information fusion adjustment; Based on the prediction results of the Kalman filter and the actual detection results, the confidence level of the target is dynamically adjusted as follows: （10）； In formula (10), The target location detected by the Yolov8L detection model. The target location predicted by the Kalman filter. For confidence level adjustment, This is the position deviation threshold; Pixel temporal variation information adjustment: By analyzing the pixel changes of the target over time, a dynamic confidence adjustment factor is constructed to adjust the confidence of the detection results. The pixel temporal change information adjustment is achieved by calculating the Euclidean distance between target pixels in consecutive frames, obtaining the average rate of change of this distance over the analysis time period N, and determining whether the target's motion is smooth and continuous, as detailed below: Pixel timing changes (11); In formula (11), It is the Euclidean distance of the target between time step t and time step t+1, and N is the length of the analyzed time series; the smaller the pixel time series change value, the smoother and more continuous the motion trajectory; conversely, the larger the pixel time series change value, the less smooth and discontinuous the trajectory. Based on the above evaluation, a dynamic confidence adjustment factor related to pixel temporal changes is constructed. This factor follows the following principle: when the trajectory is smooth and continuous, the adjustment factor is ≥1 to enhance or maintain the detection confidence; when the trajectory is not smooth or discontinuous, the adjustment factor is <1 to weaken the detection confidence. Multiply the original detection confidence level by the dynamic confidence level adjustment factor to obtain the final adjusted confidence level, as shown in formula (12): Confidence level = Confidence level · Dynamic confidence level adjustment factor (12); Scene structure information fusion adjustment: The confidence weight of targets falling into the boundary area is automatically increased using scene structure information. The area outside the boundary area is considered a danger zone, while the area inside the boundary area is considered a safe zone. Specifically: Scene structure information weight (13); In formula (13), In order to be in the danger zone, To be outside the danger zone, It is the weighting coefficient for dangerous areas, and its value is greater than 1.

7. The railway personnel intrusion detection method based on visual perception and scene structure understanding according to claim 3, characterized in that, S4 specifically includes: A lightweight LSTM network is constructed to model the typical motion patterns of targets in video sequences: low-level behavioral features include target bounding box coordinates, velocity, and acceleration; high-level behavioral patterns include walking, vaulting, crouching, and lingering. (1) Using low-level behavioral features as input, a hierarchical LSTM structure is introduced to extract and fuse high-level behavioral patterns layer by layer, and finally output the target behavior probability distribution. (2) Combine the behavioral probability distribution obtained in (1) with the target's true size and motion trend estimated by the spatial scale mapping model to correct the behavioral probability distribution; (3) Using the behavior probability distribution modified in (2) as the weight, the final behavior probability distribution is generated by weighted fusion with the visual detection confidence.

8. The railway personnel intrusion detection method based on visual perception and scene structure understanding according to claim 7, characterized in that, An improved LSTM network is created by introducing a hierarchical LSTM structure, consisting of an input layer, a feature fusion layer, LSTM units, and an output layer. Input layer: Low-level feature vectors are The high-level feature vector is For each time step t, the low-level features and high-level features are concatenated into a new feature vector: （14）； Then concatenated feature vectors As input to the LSTM input layer; Feature fusion layer: Inside the LSTM, the improved feature fusion process is as follows: Attention-weighted fusion introduces an attention mechanism to weight and fuse low-level and high-level features, enabling LSTM to automatically learn the importance of different features. Attention weight coefficients are calculated as follows: （15）； In formula (15), It is the attention weight matrix. It is the attention bias vector. Used to normalize attention weights; The fused feature vector is obtained by feature weighting and fusion calculation. : （16）； In formula (16), It is element-wise multiplication; LSTM units include input gates Forgotten Gate Candidate cell status Cell state Output gate and hidden state Among them, the hidden state Indicates the output of the LSTM unit; The output layer of the improved LSTM network uses the Softmax function to store the hidden states. Probability distribution for conversion into behavior: （17）； In formula (17), The target performs actions from time 1 to time t. The probability, It is the total number of behavior categories. and These are the weights and biases of the output layer.

9. A system for implementing a railway personnel intrusion detection method based on visual perception and scene structure understanding, characterized in that, The system includes: a data acquisition module, a scene understanding module, a dynamic confidence adjustment module, a behavior pattern modeling module, and a comprehensive decision-making module; The data acquisition module is used to acquire image data in real time; the scene understanding module includes a Yolov8L detection model and a multi-task convolutional network, which are used to identify intrusion targets and acquire boundary regions, respectively; the dynamic confidence adjustment module is used to adjust the confidence of the detection results; the behavior pattern modeling module is used to detect the probability distribution of behavior; and the comprehensive decision module is used to determine whether to trigger an intrusion alarm and issue an alarm.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images