A machine vision-based locomotive auxiliary driving pedestrian intrusion detection system

Abstract

The application discloses a locomotive auxiliary driving pedestrian intrusion limit detection system based on machine vision and relates to the technical field of railways.The system comprises a joint representation module, a time series prediction module, a dynamic risk module and a hierarchical output module.The application obtains locomotive multi-modal data, simultaneously performs geometric correction on the data, generates track plane three-dimensional point clouds and fits track geometric priors, takes the track geometric priors as position coding reference points, performs deformable sampling on the extracted pedestrian detection feature map, calculates cross-modal similarity, outputs a shared bird's eye view feature map, effectively eliminates perspective distortion in a complex scene, realizes accurate alignment of track geometry and pedestrian spatial position, provides a high-reliability spatial reference for intrusion limit judgment, and significantly improves the positioning accuracy and system robustness of pedestrian intrusion limit detection in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of railway technology, in particular to a locomotive auxiliary driving pedestrian intrusion limit detection system based on machine vision. BACKGROUND

[0002] Mine railway transportation is an important link of mine production scheduling, and undertakes the task of transferring ore, materials and products. At present, the operating mileage of enterprise-owned railway special line in China continues to grow. The locomotive has the advantages of large traction force and high transportation efficiency, and is widely used in factory, port, oil field and railway station. However, due to the fact that the infrastructure protection along the mine railway is not perfect, the management area is open, and the personnel pass frequently, the illegal intrusion of pedestrians into the rail limit has become one of the main reasons leading to transportation safety accidents. According to statistics, in the track transportation accidents, the proportion of pedestrians and staff crossing the rail area intrusion limit is very high, and the safety hidden danger is outstanding.

[0003] At present, in the complex track environment such as curved track, slope and tunnel entrance, the traditional geometric projection method based on monocular vision is seriously affected by perspective distortion: the track at the curved track is contracted in curve, the slope leads to distortion of depth scale, and the sudden change of light at the tunnel entrance further aggravates the image distortion. The horizontal distance and vertical depth error of the pedestrian relative to the track center calculated from the image are significantly enlarged, which often misjudges the pedestrians outside the safety distance as intrusion, or misses the dangerous targets close to the track, seriously restricting the accuracy and reliability of intrusion limit warning.

[0004] Therefore, the present application provides a locomotive auxiliary driving pedestrian intrusion limit detection system based on machine vision to solve the above problems. SUMMARY

[0005] The main purpose of the present application is to provide a locomotive auxiliary driving pedestrian intrusion limit detection system based on machine vision to solve the problems raised in the above background.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a locomotive auxiliary driving pedestrian intrusion limit detection system based on machine vision, which comprises a joint representation module, a time series prediction module, a dynamic risk module and a hierarchical output module. The joint representation module is used to encode the track geometry prior and pedestrian detection features in space position by using deformable attention mechanism, and generate shared bird's eye view feature map.

[0007] The time series prediction module is used to extract the 3D skeleton point sequence of the pedestrian, and output the future multi-modal trajectory probability distribution based on convolutional network. The dynamic risk module is used to construct a space-time risk function including track distance term, predicted relative distance term and predicted uncertainty term, and determine the most dangerous intrusion time and direction by solving the field gradient direction. The hierarchical output module is used for outputting hierarchical alarm signals and visual information according to the risk field peak value and the residual reaction time, and the visual information includes a pedestrian position snapshot, a predicted trajectory arrow, a risk heat map and a confidence interval.

[0008] Preferably, the joint representation module includes a data acquisition unit, a depth estimation unit and an attention fusion unit. The data acquisition unit is used for acquiring an RGB image in front of the locomotive through a monocular camera, acquiring a pitch angle, a roll angle and acceleration data of the locomotive through an inertial measurement unit, and obtaining an instantaneous speed of the locomotive through a wheel speed meter.

[0009] Preferably, the depth estimation unit receives the RGB image, outputs a depth value of each pixel, and performs geometric correction on the depth estimation result based on the pitch angle, the roll angle and the acceleration information, to generate a three-dimensional point cloud of a track plane, and further obtain track geometric priors including a track center line equation, a track gauge and a curvature radius through curve fitting.

[0010] Preferably, the attention fusion unit uses the track geometric priors as reference points for position coding, performs deformable sampling on each feature point in a pedestrian detection feature map extracted from the original RGB image by a pre-trained pedestrian detection network, and calculates a cross-modal similarity between track features and pedestrian features through a multi-head attention mechanism to output a shared bird's eye view feature map; the pedestrian detection feature map contains pedestrian confidence, boundary box offset and human key point heat value; the pedestrian detection network is a lightweight convolutional neural network, and the resolution of the bird's eye view feature map is 200x200 grids, each grid unit contains track existence probability, pedestrian joint heat, depth offset and semantic category.

[0011] Preferably, the time series prediction module includes a time series correction unit and a convolution prediction unit. The time series correction unit extracts two-dimensional joint points of pedestrians from consecutive video frames, combines the depth estimation result to project to three-dimensional space to obtain an initial three-dimensional skeleton point sequence, and then smoothes and corrects the consecutive 20 frame skeleton point sequence through a time series convolution network to output a stable three-dimensional skeleton point sequence, the three-dimensional skeleton point sequence includes three-dimensional space coordinates of at least 14 human key points; The convolution prediction unit includes a spatial convolution branch, a time series convolution branch and a fusion branch. The spatial convolution branch constructs a human pose graph with human skeleton points as graph nodes and skeleton connections as edges, and extracts spatial collaborative motion features between joints through a graph convolution network. The time series convolution branch adopts a multi-layer causal dilated convolution to extract time dimension features of the historical trajectory coordinate sequence to obtain time series features.

[0012] Preferably, the fusion branch is used to splice the spatial features with the time sequence features, and then output the future multi-modal trajectory probability distribution through a fully connected network; The multi-modal trajectory probability distribution is expressed by a Gaussian mixture model, and includes three behavior modes of straight driving, approaching a track, and sudden invasion, and respective probabilities, mean vectors, and covariance matrices.

[0013] Preferably, the dynamic risk module includes a risk construction unit and an invasion solving unit. The risk construction unit is configured to construct a space-time risk function, and the space-time risk function is equal to a sum of a first risk value, a second risk value, and a third risk value. The first risk value is a first preset weight coefficient multiplied by one divided by a sum of a track distance and a zero-prevention constant; the track distance is an Euclidean distance from any grid point in a bird's-eye view grid to a nearest track center line. The second risk value is a second preset weight coefficient multiplied by one divided by a sum of a predicted relative distance and the zero-prevention constant; the predicted relative distance is an Euclidean distance between a predicted pedestrian position and a predicted train position; the predicted pedestrian position is a pedestrian position at a future time obtained by taking a weighted average of each mode based on the multi-modal trajectory probability distribution; and the predicted train position is a train position at the future time obtained by integrating a train speed. The third risk value is a third preset weight coefficient multiplied by a predicted uncertainty variance; and the predicted uncertainty variance is obtained by sampling by a Monte Carlo rejection method. Specifically, a dropout layer is enabled when the convolution prediction unit is reasoning, T times of forward propagation are performed, and T is in a range of twenty times to fifty times; a mean value and a variance of T times of output results are calculated, and the variance is taken as the predicted uncertainty. When the predicted uncertainty exceeds a preset prediction threshold, a confidence level of ninety-five percent is adopted, and an output form of the warning time is: the most dangerous invasion time plus or minus one point nine six multiplied by a square root of the predicted uncertainty.

[0014] Preferably, the invasion solving unit obtains a local maximum point of a risk field by solving a gradient of the space-time risk function equal to zero, and the local maximum point corresponds to the most dangerous invasion time and a corresponding grid coordinate corresponds to an invasion direction.

[0015] Preferably, the hierarchical output module includes a risk calculation unit, a hierarchical unit, and a learning optimization unit. The risk calculation unit is configured to calculate a difference value based on a current time and the most dangerous invasion time, and obtain a remaining reaction time. A maximum value of the space-time risk function in all bird's-eye view grid coordinates and all predicted time ranges is obtained, and a risk field peak value is obtained. The grading unit divides the risk level based on the remaining reaction time and the risk field peak value, and the risk level includes: The first level: when the remaining reaction time is greater than the third threshold value and the risk field peak value is less than the first threshold value, the interface prompt information is output, the position and the predicted trajectory of the pedestrian are displayed, and the sound and light alarm is not triggered; The second level: when the remaining reaction time is greater than the second threshold value and less than or equal to the third threshold value, and the risk field peak value is greater than or equal to 0.3 and less than 0.7, the sound alarm and the screen highlight display are output, and the frequency of the sound alarm increases with the decrease of the remaining reaction time; The third level: when the remaining reaction time is less than the second threshold value and the risk field peak value is greater than or equal to the fourth threshold value, a brake request signal is output to the train control system, and video data and sensor data within 10 seconds before and after the event occurrence time are recorded.

[0016] Preferably, the learning optimization unit is used for online collection of difficult examples of mis-detection or missed detection scenes, and the algorithm is fine-tuned on the vehicle end through elastic weight consolidation to continuously adapt to different line environments. The elastic weight consolidation algorithm measures the importance of each parameter through the diagonal elements of the Fisher information matrix, adds a regularization term during fine-tuning, the regularization coefficient is 10 to 100, the fine-tuning trigger condition is that the cache is full for 1,000 frames or the cumulative running time is 100 hours, the fine-tuning is performed for 10 rounds, the learning rate is 1 / 10 of the initial value, and when the accuracy of the updated model on the verification set decreases by no more than 2%, the old model is replaced, and the old model is retained as a rollback version.

[0017] The present application has the following beneficial effects: 1. In the present application, multi-modal data of a locomotive is acquired, the data is geometrically corrected, track plane three-dimensional point clouds are generated, and track geometric priors are fitted, the extracted pedestrian detection feature map is deformable sampled with the track geometric prior as a position coding reference point, cross-modal similarity is calculated, a shared bird's eye view feature map is output, perspective distortion in a complex scene is effectively eliminated, accurate alignment of track geometry and pedestrian spatial position is realized, a high-reliability spatial reference for intrusion judgment is provided, and the positioning accuracy and system robustness of pedestrian intrusion detection in a complex environment are significantly improved.

[0018] 2. In the present application, by extracting the two-dimensional nodes of pedestrians, combining depth estimation back projection into three-dimensional space, and obtaining stable three-dimensional skeleton point sequence through temporal convolution smoothing, the shaking and occlusion error are reduced, at the same time, the human body posture graph is constructed, and the spatial coordination features are extracted by using Chebyshev convolution, and the historical trajectory temporal features are extracted by using multi-layer causal inflation convolution, and after fusion, the multi-modal trajectory probability distribution is output, the probabilistic modeling of pedestrian motion intention is realized, the pertinence of early warning is improved, the space-time risk function is constructed, the most dangerous invasion time and direction are determined by gradient solution, the early warning interval with confidence level is output by combining Monte Carlo method, the invasion risk level is quantified, and the reliability of the system in uncertain scene is enhanced.

[0019] 3. In the present application, according to the remaining reaction time and the peak value of the risk field, three levels of early warning are divided: the first level interface prompts, the second level sound and light alarms and the frequency increases with the decrease of the remaining time, the third level outputs the brake request and records the event data, realizes the risk grading response and dynamic warning; the learning optimization unit collects misjudgment, missed difficult cases online, and the algorithm is consolidated by elastic weight, the parameter importance is measured by the diagonal element of the Hill information matrix, a regularization term is added, and the micro-adjustment is triggered when the cache is full or the cumulative running time is one hundred hours, the model is updated when the verification set accuracy decreases by no more than two percent, and the old version is reserved as rollback, realizing the vehicle self-supervised continuous learning, avoiding forgetting and without manual annotation. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The flowchart of the locomotive auxiliary driving pedestrian invasion detection system based on machine vision provided by the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0022] The present application provides a technical solution: a locomotive auxiliary driving pedestrian invasion detection system based on machine vision, the system comprising a joint representation module, a time series prediction module, a dynamic risk module and a grading output module. The joint representation module is used to encode the track geometry priori and the pedestrian detection features in spatial position by using deformable attention mechanism, to generate shared bird's eye view feature map.

[0023] The time series prediction module is used to extract the 3D skeleton point sequence of the pedestrian, and output the future multi-modal trajectory probability distribution based on the convolution network. The dynamic risk module is used for constructing a space-time risk function comprising a track distance item, a predicted relative distance item and a predicted uncertainty item, and determining a most dangerous invasion time and direction by solving a field gradient direction; The hierarchical output module is used for outputting hierarchical alarm signals and visual information according to a risk field peak value and a remaining reaction time, and the visual information comprises a pedestrian position snapshot, a predicted trajectory arrow, a risk heat map and a confidence interval.

[0024] The joint representation module comprises a data acquisition unit, a depth estimation unit and an attention fusion unit. The data acquisition unit is used for acquiring an RGB image in front of the locomotive through a monocular camera, acquiring a pitch angle, a roll angle and acceleration data of the locomotive through an inertial measurement unit, and acquiring an instantaneous speed of the locomotive through a wheel speed meter.

[0025] The depth estimation unit receives the RGB image, outputs a depth value of each pixel, and performs geometric correction on the depth estimation result based on the pitch angle, the roll angle and the acceleration information, to generate a three-dimensional point cloud of a track plane, and then obtains track geometry priors including a track center line equation, a track gauge and a curvature radius through curve fitting.

[0026] Specifically, the depth estimation unit adopts a FastDepth model, inputs a single RGB image, and outputs a dense depth map; network parameters are obtained through pre-training and fine-tuning, and each pixel depth value is directly used as a network output feature map channel value during inference; The attention fusion unit is used for taking the track geometry priors as reference points of position encoding, performing deformable sampling on each feature point in a pedestrian detection feature map extracted from the original RGB image by a pre-trained pedestrian detection network, and calculating a cross-modal similarity between track features and pedestrian features through a multi-head attention mechanism to output a shared bird's eye view feature map; the pedestrian detection feature map comprises a pedestrian confidence, a bounding box offset and a human key point heat value; the pedestrian detection network is a lightweight convolutional neural network, and the resolution of the bird's eye view feature map is 200x200 grids, each grid unit comprising a track existence probability, a pedestrian joint heat, a depth offset and a semantic category.

[0027] Specifically, the lightweight convolutional neural network is a YOLOv8-Nano model, which is an existing mature technology and will not be described here in detail, wherein the training set comprises more than ten thousand railway scene images labeled with pedestrian bounding boxes, and the images cover daytime, nighttime, rain and fog, tunnel entrance and curved road scenes; The time series prediction module comprises a time series correction unit and a convolutional prediction unit. The timing correction unit obtains an initial three-dimensional skeleton point sequence by projecting a two-dimensional skeleton joint extracted from consecutive video frames to a three-dimensional space in combination with a depth estimation result, and then smoothes the consecutive 20 frame skeleton point sequences through a timing convolution network to output a stable three-dimensional skeleton point sequence, the three-dimensional skeleton point sequence including three-dimensional space coordinates of at least 14 human body key points; The convolution prediction unit includes a spatial convolution branch, a timing convolution branch and a fusion branch; The spatial convolution branch constructs a human body pose graph with human body skeleton points as graph nodes and skeleton connections as edges, and extracts spatial collaborative motion features between joints through a graph convolution network. Specifically, the graph convolution network is a Chebyshev graph convolution, the convolution kernel order of which is 2, and the output channel number of each layer is 64, which is used to weight and aggregate the features of adjacent joints to obtain a spatial feature vector of each joint. The timing convolution branch uses a multi-layer causal dilated convolution to extract time dimension features of the historical trajectory coordinate sequence to obtain timing features.

[0028] Specifically, the multi-layer causal dilated convolution is a four-layer causal dilated convolution, the dilated rates of which are 1, 2, 4 and 8 respectively, the convolution kernel size is 3, the step is 1, and the receptive field covers the trajectory information in the past 2 seconds; each layer is followed by a batch normalization layer and a linear rectifier activation function The fusion branch is used to concatenate the spatial features and the timing features, and then output the future multi-modal trajectory probability distribution through a fully connected network. Specifically, the concatenation method is: for each time step, the spatial feature vector and the timing feature are concatenated at the head and tail in the channel dimension to form a fusion feature vector; then a two-layer fully connected network is used to output the future multi-modal trajectory probability distribution; the hidden layer dimension of the two-layer fully connected network is 128, and the output layer dimension is 3 times the number of parameters of each mode. The multi-modal trajectory probability distribution is represented by a mixture Gaussian model, which includes three behavior modes of straight, close to the track and sudden intrusion, and the corresponding probability, mean vector and covariance matrix.

[0029] Specifically: for each behavior mode, the mean vector of the predicted position at multiple time points in the future 1 to 2 seconds is output, the mean vector being a two-dimensional coordinate with a unit of meters; For each behavior mode, the covariance matrix of the predicted position is output, the covariance matrix being a 2x2 positive definite matrix represented by diagonal elements and off-diagonal elements. The dynamic risk module includes a risk construction unit and an intrusion solving unit. The risk construction unit is used to construct a spatio-temporal risk function, which is equal to the sum of a first risk value, a second risk value and a third risk value. The first risk value is a first preset weight coefficient multiplied by one divided by the sum of the track distance and the anti-zero constant; the track distance is the Euclidean distance from any grid point in the bird's eye view grid to the nearest track center line; The second risk value is a second preset weight coefficient multiplied by one divided by the sum of the predicted relative distance and the anti-zero constant; the predicted relative distance is the Euclidean distance between the predicted pedestrian position and the predicted locomotive position; the predicted pedestrian position is the pedestrian position at the future time obtained by taking the weighted average of each mode based on the multi-modal trajectory probability distribution; the predicted locomotive position is the locomotive position at the future time obtained by integrating the locomotive speed; The third risk value is a third preset weight coefficient multiplied by the predicted uncertainty variance; the predicted uncertainty variance is obtained by Monte Carlo rejection sampling; Specifically, a dropout layer is enabled during inference of the convolution prediction unit, and T times of forward propagation are performed, and the value of T is in the range of twenty to fifty times; the mean and variance of T times of output results are calculated, and the variance is taken as the prediction uncertainty; When the prediction uncertainty exceeds the preset prediction threshold, a confidence level of 95% is used, and the output form of the warning time is: the most dangerous intrusion time plus or minus 1.96 multiplied by the square root of the prediction uncertainty.

[0030] The prediction threshold is obtained by offline calibration or online adaptive adjustment; That is, the warning time interval is represented as: ; In the formula, represents the most dangerous intrusion time, represents the predicted uncertainty variance, represents the double-sided quantile corresponding to the 95% confidence level; Wherein, the space-time risk function is specifically represented as: ; Wherein, represents the coordinates in the bird's eye view grid, represents the current time, represents the Euclidean distance from the grid point to the nearest track center line, represents the pedestrian position at the future time predicted by the multi-modal trajectory probability distribution, represents the locomotive position at the future time obtained by integrating the locomotive speed, represents the uncertainty variance of the predicted position, 、 and represent the preset weight coefficients, and the sum is 1; Specifically, the formula for predicting pedestrian location is as follows: ; In the formula, Representing the The modality at time... The predicted position mean vector, Representing the The weights of each modality are obtained by normalizing the output layer of the fully connected network using the Softmax function. Their values ​​change dynamically with the input scene and are not preset constants. Represents the number of modes.

[0031] The intrusion solving unit obtains the local maximum point of the risk field by solving the gradient of the spatiotemporal risk function to zero. The time corresponding to this local maximum point is the most dangerous intrusion time, and the corresponding grid coordinates are the intrusion direction.

[0032] The hierarchical output module includes a risk calculation unit, a hierarchical unit, and a learning optimization unit. The risk calculation unit is used to calculate the difference between the current time and the most dangerous intrusion time to obtain the remaining response time; Obtain the maximum value of the spatiotemporal risk function across all bird's-eye view grid coordinates and all prediction time ranges to obtain the peak value of the risk field; The risk level is classified based on remaining reaction time and peak risk field, including: Level 1: When the remaining reaction time is greater than the third threshold and the peak value of the risk field is less than the first threshold, the output interface will display the pedestrian's location and predicted trajectory, without triggering the audible and visual alarms. Level 2: When the remaining reaction time is greater than the second threshold and less than or equal to the third threshold, and the peak value of the risk field is greater than or equal to 0.3 and less than 0.7, an audible alarm is output and the screen is highlighted. The frequency of the audible alarm increases as the remaining reaction time decreases. Level 3: When the remaining reaction time is less than the second threshold and the peak value of the risk field is greater than or equal to the fourth threshold, a braking request signal is output to the train control system, and video data and sensor data for ten seconds before and after the event occur are recorded.

[0033] The specific values ​​of the first threshold, second threshold, third threshold, and fourth threshold are not fixed and can be calibrated by the system manufacturer based on the locomotive braking distance, sensor refresh rate, driver reaction time, and speed limit requirements of the operating route. The learning optimization unit is used to collect difficult examples of false detection or missed detection scenarios online, and the model is fine-tuned on the vehicle side through the elastic weight consolidation algorithm to continuously adapt to different road environments; The elastic weight consolidation algorithm measures the importance of each parameter by the diagonal elements of the Fisher information matrix, adds a regularization term during fine-tuning, the regularization coefficient is ten to one hundred, the fine-tuning trigger condition is that the cache is full of one thousand frames or accumulates one hundred hours of running, fine-tuning is ten rounds, and the learning rate is one tenth of the initial value; the updated model replaces the old model when the accuracy on the validation set decreases by no more than two percent, and the old model is retained as a rollback version.

[0034] Specifically, the importance of the calculation parameter is: ; In the formula, represents the importance of the th parameter, represents the expected operation, and indicates the average value of the sample in the data distribution , represents the input data, represents the data distribution of the old task, represents a vector composed of all trainable parameters of the model, represents the label output by the model, represents the th parameter, represents the conditional probability of the model output under the condition of given input and model parameters ; When fine-tuning, a regularization term is added, and the regularization coefficient is ten to one hundred, so the regularization loss function is represented as: ; In the formula, represents the cross-entropy loss or mean square error loss on the new difficult example data according to the task type, represents the old parameter before fine-tuning, represents the regularization coefficient; It should be noted that in this paper, relational terms such as first, second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes" "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0035] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A machine vision-based pedestrian encroachment detection system for vehicle assisted driving, characterized in that, The system includes a joint characterization module, a time series prediction module, a dynamic risk module, and a hierarchical output module; The joint representation module is used to encode the spatial position of the track geometry prior and pedestrian detection features using a deformable attention mechanism to generate a shared bird's-eye view feature map. The temporal prediction module is used to extract the 3D skeleton point sequence of pedestrians and output the probability distribution of future multimodal trajectories based on the convolutional network. The dynamic risk module is used to construct a spatiotemporal risk function that includes an orbital distance term, a predicted relative distance term, and a predicted uncertainty term, and determines the most dangerous intrusion time and location by solving for the field gradient direction; The graded output module is used to output graded alarm signals and visualization information based on the peak value of the risk field and the remaining reaction time. The visualization information includes pedestrian location snapshots, predicted trajectory arrows, risk heat maps, and confidence intervals.

2. The locomotive assisted driving pedestrian intrusion detection system according to claim 1, characterized in that, The joint characterization module includes a data acquisition unit, a depth estimation unit, and an attention fusion unit; The data acquisition unit is used to acquire RGB images of the locomotive from the front using a monocular camera, and to acquire the locomotive's pitch angle, roll angle, and acceleration data using an inertial measurement unit, and to obtain the locomotive's instantaneous speed using a wheel speed meter.

3. The locomotive assisted driving pedestrian intrusion detection system according to claim 2, characterized in that, The depth estimation unit receives RGB images, outputs the depth value of each pixel, and performs geometric correction on the depth estimation results based on pitch angle, roll angle and acceleration information to generate a three-dimensional point cloud of the track plane. Then, the track geometric prior is obtained through curve fitting. The track geometric prior includes the track centerline equation, track gauge and radius of curvature.

4. The locomotive assisted driving pedestrian intrusion detection system according to claim 2, characterized in that, The attention fusion unit uses the track geometry prior as a reference point for position encoding, performs deformable sampling on each feature point in the pedestrian detection feature map extracted from the original RGB image by the pre-trained pedestrian detection network, and calculates the cross-modal similarity between track features and pedestrian features through a multi-head attention mechanism, outputting a shared bird's-eye view feature map; the pedestrian detection feature map includes pedestrian confidence, bounding box offset, and human keypoint heat values; the pedestrian detection network is a lightweight convolutional neural network, and the resolution of the bird's-eye view feature map is 200x200 grids, with each grid cell containing track existence probability, pedestrian joint heat value, depth offset, and semantic category.

5. The locomotive assisted driving pedestrian intrusion detection system according to claim 1, characterized in that, The temporal prediction module includes a temporal correction unit and a convolutional prediction unit; The temporal correction unit extracts two-dimensional key points of pedestrians from continuous video frames, back-projects the depth estimation results to three-dimensional space to obtain an initial three-dimensional skeleton point sequence, and then smooths and corrects the skeleton point sequence of 20 consecutive frames through a temporal convolutional network to output a stable three-dimensional skeleton point sequence. The three-dimensional skeleton point sequence includes the three-dimensional spatial coordinates of at least 14 human key points. The convolutional prediction unit includes a spatial convolutional branch, a temporal convolutional branch, and a fusion branch; The spatial convolutional branch constructs a human posture map using human skeleton points as graph nodes and skeleton connections as edges, and extracts spatial cooperative motion features between joints through a graph convolutional network. The temporal convolution branch employs multi-layer causal dilated convolution to extract temporal features from the historical trajectory coordinate sequence, thereby obtaining temporal features.

6. The locomotive assisted driving pedestrian intrusion detection system according to claim 5, characterized in that, The fusion branch is used to concatenate spatial features and temporal features, and then output the probability distribution of future multimodal trajectories through a fully connected network; The multimodal trajectory probability distribution is represented by a Gaussian mixture model, which includes three behavior modes: straight, approaching the track, and sudden intrusion, as well as their corresponding probabilities, mean vectors, and covariance matrices.

7. The locomotive assisted driving pedestrian intrusion detection system according to claim 1, characterized in that, The dynamic risk module includes a risk construction unit and an intrusion solution unit; The risk construction unit is used to construct a spatiotemporal risk function, which is equal to the sum of a first risk value, a second risk value, and a third risk value. The first risk value is the sum of the first preset weighting coefficient multiplied by one and divided by the track distance and the zero-prevention constant; the track distance is the Euclidean distance from any grid point in the bird's-eye view grid to the nearest track centerline; The second risk value is the second preset weighting coefficient multiplied by one and divided by the sum of the predicted relative distance and the zero-prevention constant; the predicted relative distance is the Euclidean distance between the predicted pedestrian position and the predicted locomotive position; the predicted pedestrian position is the future pedestrian position obtained by taking the weighted average of each mode based on the multimodal trajectory probability distribution; the predicted locomotive position is the future locomotive position obtained by integrating the locomotive speed. The third risk value is the third preset weighting coefficient multiplied by the prediction uncertainty variance; the prediction uncertainty variance is obtained by Monte Carlo dropout sampling. Specifically, a dropout layer is enabled during inference of the convolutional prediction unit, and T forward propagations are performed, with T ranging from twenty to fifty times; the mean and variance of the T output results are calculated, and the variance is used as the prediction uncertainty. When the prediction uncertainty exceeds the preset prediction threshold, a 95% confidence level is adopted, and the output form of the warning time is: the most dangerous intrusion time plus or minus 1.96 times the square root of the prediction uncertainty.

8. The locomotive assisted driving pedestrian intrusion detection system according to claim 7, characterized in that, The intrusion solving unit obtains the local maximum point of the risk field by solving the gradient of the spatiotemporal risk function to zero. The time corresponding to the local maximum point is the most dangerous intrusion time, and the corresponding grid coordinates are the intrusion direction.

9. A machine vision-based pedestrian encroachment detection system for vehicle assisted driving according to claim 1, characterized in that, The hierarchical output module includes a risk calculation unit, a hierarchical unit, and a learning optimization unit; The risk calculation unit is used to calculate the difference between the current time and the most dangerous intrusion time to obtain the remaining reaction time. Obtain the maximum value of the spatiotemporal risk function across all bird's-eye view grid coordinates and all prediction time ranges to obtain the peak value of the risk field; The classification unit categorizes risk levels based on remaining reaction time and peak risk field, including: Level 1: When the remaining reaction time is greater than the third threshold and the peak value of the risk field is less than the first threshold, the output interface will display the pedestrian's location and predicted trajectory, without triggering the audible and visual alarms. Level 2: When the remaining reaction time is greater than the second threshold and less than or equal to the third threshold, and the peak value of the risk field is greater than or equal to 0.3 and less than 0.7, an audible alarm is output and the screen is highlighted. The frequency of the audible alarm increases as the remaining reaction time decreases. Level 3: When the remaining reaction time is less than the second threshold and the peak value of the risk field is greater than or equal to the fourth threshold, a braking request signal is output to the train control system, and video data and sensor data for ten seconds before and after the event occur are recorded.

10. A machine vision-based pedestrian encroachment detection system for vehicle assisted driving according to claim 9, characterized in that, The learning optimization unit is used to collect difficult examples of false detection or missed detection scenarios online, and to fine-tune the model on the vehicle end through the elastic weight consolidation algorithm to continuously adapt to different road environments; The elastic weight consolidation algorithm measures the importance of each parameter through the diagonal elements of the Fisher information matrix. During fine-tuning, a regularization term is added with a regularization coefficient of 10 to 100. The fine-tuning is triggered when the cache is full of 1,000 frames or the cumulative running time is 100 hours. Ten rounds of fine-tuning are performed, and the learning rate is one-tenth of the initial value. When the accuracy of the updated model on the validation set decreases by no more than 2%, the old model is replaced, and the old model is retained as a rollback version.

Images