Railway locomotive and self-wheel running safety management and control system based on video AI

Abstract

The present application relates to the technical field of railway safety monitoring, in particular to a railway locomotive and self-wheel operation safety control system based on video AI, comprising a video acquisition module, a video processing module, a posture analysis module, a behavior judgment module and a evidence generation module. The video acquisition module is arranged with a terminal with zooming and low-illumination capability in the key operation area and equipment operation interval to obtain the on-site operation video stream; the video processing module performs frame extraction and timestamp synchronization on the video stream to generate a video frame sequence associated with the equipment operation state; the posture analysis module inputs the video frame sequence into a pre-trained model, extracts key point coordinates and calculates the vector difference of the car body deflection angle and the wheel running direction; the behavior judgment module identifies the illegal action segment in combination with the preset regulations; and the evidence generation module traces back the original video to generate an evidence data package containing context video and posture data log. The system improves the precision and automation level of safety control.

Description

Technical Field

[0001] This invention relates to the field of railway safety monitoring technology, and in particular to a safety control system for railway locomotives and self-propelled vehicles based on video AI. Background Technology

[0002] The safe operation of railway locomotives and self-propelled equipment is the core link of railway transportation safety. At present, existing safety control technologies mostly rely on manual monitoring combined with traditional video recording methods. By deploying ordinary video acquisition equipment in key areas of railway lines to obtain on-site operation videos, staff can observe the images in real time to identify abnormal equipment operation postures and violations of operating rules. Some systems can only realize the functions of video storage and playback for post-event traceability.

[0003] Existing technical solutions have shortcomings. Manual monitoring is affected by factors such as staff fatigue and lack of concentration, making it impossible to achieve 24-hour uninterrupted and accurate monitoring. It is prone to missed or false diagnoses and cannot accurately quantify and analyze the operating posture of railway locomotives and self-propelled equipment, making it difficult to capture subtle deviations in vehicle body deflection and wheel direction. Furthermore, existing video surveillance systems can only record on-site footage. When violations occur, they cannot quickly correlate the equipment posture data corresponding to the violations, nor can they efficiently trace the context before and after the violations, failing to form a complete chain of evidence for violations. This causes numerous inconveniences for the identification of violations and subsequent rectification.

[0004] Existing technologies cannot accurately quantify and identify the posture of railway locomotives and self-propelled equipment, making it difficult to accurately capture violations that deviate from standard operating postures. Furthermore, after a violation occurs, there is a lack of complete evidence that simultaneously includes posture data and contextual video, which fails to meet the actual needs of automated, precise, and standardized railway safety management. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a safety control system for railway locomotives and self-propelled vehicles based on video AI.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a safety control system for railway locomotives and self-propelled vehicles based on video AI, comprising:

[0007] The video acquisition module deploys multiple video acquisition terminals with zoom and low-light capabilities in key operating areas and mobile equipment operating zones along the target railway line to continuously acquire on-site operation video streams including railway locomotives and self-propelled equipment.

[0008] The video processing module performs frame extraction and timestamp synchronization processing on the acquired on-site operation video stream to generate a video frame sequence associated with the operating status of the railway locomotive and self-propelled equipment.

[0009] The attitude analysis module inputs the video frame sequence into a pre-trained locomotive attitude recognition model, extracts the coordinates of key points of the railway locomotive and self-propelled equipment in the image, and calculates the vector difference between the vehicle body deflection angle and the wheel travel direction.

[0010] The behavior determination module inputs the vector difference between the vehicle body deflection angle and the wheel travel direction into the behavior compliance determination engine, and, in combination with the preset work procedure constraints, identifies the non-compliant action segments that deviate from the standard working posture.

[0011] The evidence generation module, based on the timestamp of the occurrence of the violation action segment, traces back the original surveillance video corresponding to the video frame sequence, extracts and packages it to generate a violation event evidence data package, which includes context video segments before and after the violation and posture data logs.

[0012] As a further aspect of the present invention, the video frame sequence is input into a pre-trained locomotive attitude recognition model to extract the coordinates of key points of the railway locomotive and self-propelled equipment in the image, and to calculate the vector difference between the vehicle body deflection angle and the wheel travel direction, including:

[0013] The convolutional neural network is used to perform semantic segmentation on each frame of the video frame sequence to identify the outline of the railway locomotive and self-propelled equipment, the bogie assembly, and the location area of ​​the wheel axle connection point;

[0014] Within the frame contour area, the coordinates of the center points at both ends of the vehicle body are located using a key point regression algorithm, and the angle between the line connecting the two points and the horizontal baseline is calculated and defined as the vehicle body deflection angle.

[0015] Within the location area of ​​the wheel axle connection point, the displacement of the center pixel of the same wheel in two adjacent frames is tracked, and the direction of travel and displacement vector of the wheel per unit time are calculated, which is defined as the vector of the direction of travel of the wheel.

[0016] The vector corresponding to the vehicle body deflection angle is geometrically compared with the vector of the wheel travel direction, and the calculated direction deviation value is used as the vector difference.

[0017] The vehicle body deflection angle, the vector of the wheel travel direction, and the vector difference are packaged into an attitude feature vector for subsequent compliance determination.

[0018] The steps for constructing the locomotive attitude recognition model include:

[0019] Training video images containing different types of railway locomotives and self-propelled equipment under various working conditions were collected. The chassis outline, bogie components and wheel axle connection points in the training video images were manually labeled to generate a training dataset with location labels.

[0020] A semantic segmentation model backbone network based on a deep convolutional neural network was constructed, and a feature pyramid structure was added to fuse multi-scale features for processing training video images of different resolutions.

[0021] Two branch networks are connected in parallel at the output of the backbone network of the semantic segmentation model. The first branch network outputs the vehicle body contour segmentation map, and the second branch network outputs the heat map of key points of the bogie and wheel axle.

[0022] The semantic segmentation model backbone network and two branch networks are jointly trained end-to-end using the training dataset. The training loss function is a weighted sum of the cross-entropy loss of the vehicle body contour and the mean square error loss of the key point heatmap.

[0023] After the model training is completed, the network parameters are frozen, and the locomotive attitude recognition model is deployed to the attitude analysis module for real-time processing of the video frame sequence.

[0024] As a further aspect of the present invention, the vector difference between the vehicle body deflection angle and the wheel travel direction is input to the behavior compliance determination engine. Combined with preset work procedure constraints, it identifies violation action segments that deviate from the standard working posture, including:

[0025] Retrieve the operating procedure constraints that match the type of operation currently being performed by the railway locomotive and self-propelled equipment. The operating procedure constraints define the maximum allowable deflection threshold of the car body and the tolerance range of the vector difference under specific working conditions.

[0026] The vehicle body deflection angle in the attitude feature vector is compared with the maximum deflection threshold of the vehicle body. If the threshold is exceeded, it is marked as a deflection over-limit event.

[0027] The vector difference in the attitude feature vector is compared with the vector difference tolerance interval. If it falls outside the tolerance interval, it is marked as a travel direction inconsistency event.

[0028] Logical combination judgment is performed on the deflection exceeding the limit event and the inconsistency of the direction of travel event in the continuous time series to identify the composite violation state that simultaneously satisfies the abnormal vehicle posture and the abnormal direction of travel.

[0029] Record the start and end timestamps of the identified single violation event or complex violation status to generate a time index of the violation action segment.

[0030] As a further aspect of the present invention, it also includes:

[0031] The risk detection module acquires real-time monitoring images from video acquisition terminals deployed on both sides of railway crossings, performs background modeling and foreground extraction on the images, and separates moving pedestrians, non-motorized vehicles, and motor vehicles.

[0032] The extracted moving targets are classified and identified using a deep learning target classification model, distinguishing between motor vehicles, non-motor vehicles and pedestrians, and obtaining their position coordinates and movement trajectories in the image.

[0033] Based on the physical width and clearance parameters of the railway crossing, a three-dimensional spatial bounding box for the intrusion warning area is set, and the three-dimensional spatial bounding box extends to a safe distance outside the track centerline.

[0034] The system monitors the spatial relationship between the trajectory of the moving target and the three-dimensional bounding box in real time. Once the intersection of the target trajectory and the three-dimensional bounding box is detected, the crossing intrusion alarm signal is immediately triggered.

[0035] The intrusion alarm signal at the crossing is associated with the time index of the violation action segment to generate a comprehensive safety risk event record that includes personnel intrusion and equipment violations.

[0036] As a further aspect of the present invention, a three-dimensional spatial bounding box for the intrusion warning area is set based on the physical width and clearance parameters of the railway crossing, including:

[0037] Obtain the CAD design drawings or measured data of the railway crossing, and analyze the parameters of the inner rail spacing, ballast slope and the installation height of the crossing railing.

[0038] The effective passage width of the crossing is calculated based on the inner spacing of the rails, and a preset safety redundancy distance is extended outward to determine the horizontal cross-sectional dimensions of the three-dimensional spatial bounding box.

[0039] The vertical drop from the rail surface to the ground is calculated based on the slope of the ballast side slope, and the vertical height range of the three-dimensional spatial enclosure box is determined in combination with the installation height of the level crossing railing.

[0040] By combining the horizontal cross-sectional dimensions with the vertical height range, a trapezoidal columnar spatial model that conforms to the actual terrain of the crossing is constructed, serving as the three-dimensional spatial bounding box.

[0041] The trapezoidal cylindrical spatial model is projected onto the video frame coordinate system to generate a two-dimensional polygonal detection area for real-time detection.

[0042] As a further aspect of the present invention, it also includes:

[0043] The cargo status monitoring module uses a dedicated cargo loading recognition model to detect the material accumulation outline inside the carriage or on the flatbed of the self-propelled equipment in the video footage.

[0044] Extract the vertex coordinates of the material accumulation contour, and quantify the current cargo loading rate by calculating the ratio of the contour area to the area of ​​the carriage floor.

[0045] Simultaneously, detect whether there are obvious overhanging protrusions at the edge of the material accumulation contour. If so, mark it as an off-center loading or over-limit state.

[0046] The cargo loading rate is compared with a preset rated loading rate threshold. If it is lower than the lower limit or higher than the upper limit, an abnormal loading event is generated.

[0047] The loading anomaly event is fused with the time index of the violation action segment to generate a comprehensive violation record that includes the abnormal loading status.

[0048] As a further aspect of the present invention, the vertex coordinates of the material accumulation contour are extracted, and the current cargo loading rate is quantified by calculating the ratio of the contour area to the area of ​​the carriage floor, including:

[0049] The material accumulation contour is smoothed at the edges to remove minor noise interference caused by lighting and shadows;

[0050] The minimum bounding polygon of the processed contour is calculated using the convex hull algorithm, and the set of vertex coordinates of the polygon is obtained.

[0051] Based on the set of vertex coordinates, the actual pixel area of ​​the material accumulation contour is calculated using the shoelace formula.

[0052] The pixel area of ​​the carriage floor is obtained from the pre-calibrated carriage floor template and used as the reference area.

[0053] The percentage value obtained by dividing the actual pixel area of ​​the material stacking outline by the pixel area of ​​the carriage floor is the cargo loading rate.

[0054] As a further aspect of the present invention, it also includes:

[0055] The thermal imaging verification module simultaneously enables infrared thermal imaging video recording on the video acquisition terminals corresponding to the high-heat-generating components of the railway locomotive diesel engine compartment, traction motor, and braking system.

[0056] The infrared thermal imaging video stream is subjected to temperature threshold mapping processing to convert the grayscale image into a pseudo-color temperature heatmap;

[0057] In the pseudo-color temperature heat map, hot spots with temperatures higher than a preset warning threshold are automatically identified, and their temperature change trend curves are recorded.

[0058] Spatially register the location information of the heat-generating hotspot area with the corresponding device component location in the visible light video stream to identify the specific name of the overheated component;

[0059] The temperature change trend curve is correlated with the time index of the violation action segment to generate an early warning analysis report containing the risk of mechanical overheating.

[0060] As a further aspect of the present invention, the location information of the heat-generating hotspot area is spatially registered with the corresponding device component location in the visible light video stream to identify the specific name of the overheated component, including:

[0061] A set of known key points of equipment components are selected as reference anchor points, and the reference anchor points are manually or automatically calibrated in visible light video frames and infrared thermal imaging video frames, respectively.

[0062] The affine transformation matrix between the visible light video frame and the infrared thermal imaging video frame is calculated based on the coordinates of the anchor point in the visible light video frame and the coordinates of the corresponding anchor point in the infrared thermal imaging video frame.

[0063] Using the affine transformation matrix, the coordinates of the heat-generating hotspot region in the infrared thermal imaging video frame are mapped to the coordinate system of the visible light video frame;

[0064] In visible light video frames, based on the mapped coordinate positions, the device component labels corresponding to the coordinate positions are found, thereby determining the specific component to which the heat hotspot area belongs;

[0065] The identified component names are linked to their corresponding temperature data and stored in the equipment health status database.

[0066] As a further aspect of the present invention, the construction steps of the behavior compliance determination engine include:

[0067] The standard operating procedure document for railway locomotives and self-propelled equipment is analyzed and digitized, and the car body attitude constraints corresponding to different types of operation tasks are extracted. The car body attitude constraints include the car body deflection angle threshold, the angle threshold between the wheel travel direction and the car body centerline, and the time window when the threshold takes effect.

[0068] Based on the vehicle body attitude constraints, a rule knowledge base is constructed. The rule knowledge base is indexed by the task type and stores multiple sets of production rules consisting of conditions and conclusions.

[0069] Construct a Bayesian inference network, wherein the nodes of the Bayesian inference network correspond to the conditions and conclusions in the production rules, and calculate the conditional probability distribution between nodes based on historical violation event data;

[0070] The attitude feature vector output by the attitude analysis module is used as real-time input, first undergoes deterministic rule matching through the rule knowledge base, and then is input into the Bayesian inference network for probabilistic risk inference.

[0071] The matching results from the rule knowledge base are integrated with the inference results from the Bayesian inference network to output a judgment result containing the violation type, confidence level, and time index. The judgment result is used by the behavior judgment module to generate the time index of the violation action segment.

[0072] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0073] The video frame sequence corresponding to the railway locomotive and self-propelled equipment is input into a pre-trained locomotive attitude recognition model. The key point coordinates of the equipment in the video frame are extracted, and the vector difference between the vehicle body deflection angle and the wheel travel direction is calculated. Through the pre-trained locomotive attitude recognition model, key features of the equipment in the video frame can be accurately captured. By extracting the key point coordinates and calculating the vector difference, a quantitative analysis of the equipment's operating attitude can be achieved. This model can capture subtle deviations in vehicle body deflection and wheel travel direction that cannot be identified by conventional manual monitoring and traditional video surveillance. It eliminates reliance on manual observation and avoids missed or incorrect judgments due to subjective human judgment or fatigue, enabling accurate capture of abnormal equipment operating attitudes and making attitude recognition more objective and accurate.

[0074] The vector difference between the vehicle body deflection angle and the wheel travel direction is input into the behavior compliance judgment engine. Combined with preset operating procedure constraints, it identifies illegal action segments and, based on the timestamp of the illegal action segment, traces back the original monitoring video, extracts and packages it to generate a violation event evidence data package containing video segments before and after the violation, as well as posture data logs. Through the behavior compliance judgment engine, illegal action segments can be accurately identified based on quantified posture data, eliminating the need for manual review of massive amounts of video footage, significantly improving the efficiency of violation identification and overcoming the limitations of traditional video surveillance, which can only passively record and cannot actively identify violations. Simultaneously, by tracing back the original recording through timestamps and packaging the evidence data package containing contextual video and posture data logs, a complete chain of evidence for the violation event can be formed. This solves the problem of traditional technologies only providing single video segments and lacking posture data support, making the identification of violation events more evidence-based. Furthermore, the contextual video segments can clearly reconstruct the complete process of the violation, and the posture data logs can accurately present the equipment status at the time of the violation, meeting the need for standardized violation evidence collection in safety management. Attached Figure Description

[0075] Figure 1 This is a timing diagram of the video AI-based railway locomotive and self-propelled vehicle safety control system described in this invention.

[0076] Figure 2 A flowchart for determining behavioral compliance and identifying segments of illegal actions;

[0077] Figure 3 A flowchart for risk detection and comprehensive safety risk events at level crossings;

[0078] Figure 4 A graph showing the continuous monitoring of load rate changes for self-propelled railway equipment;

[0079] Figure 5 This is a graph showing the trend of the severity of an event over time. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0081] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0082] See Figure 1In the key operational areas and mobile equipment operating zones of the target railway line, multiple video acquisition terminals with zoom and low-light capabilities are deployed. These terminals continuously acquire on-site operational video streams containing railway locomotives and self-propelled equipment. The video processing module performs frame extraction and timestamp synchronization on the acquired on-site operational video streams, generating a video frame sequence associated with the operating status of the railway locomotives and self-propelled equipment. The attitude analysis module inputs the video frame sequence into a pre-trained locomotive attitude recognition model, extracts the coordinates of key points of the railway locomotives and self-propelled equipment in the frame, and calculates the vector difference between the vehicle body deflection angle and the wheel travel direction. The behavior determination module inputs the vector difference between the vehicle body deflection angle and the wheel travel direction into the behavior compliance determination engine. This engine, combined with preset operational procedure constraints, identifies violation action segments that deviate from the standard operating posture. The evidence generation module, based on the timestamp of the violation action segment, traces back the original monitoring video corresponding to the video frame sequence, extracts and packages it to generate a violation event evidence data package. The violation event evidence data package includes contextual video segments before and after the violation, as well as attitude data logs.

[0083] In one embodiment of the present invention, the attitude analysis module inputs a video frame sequence into a pre-trained locomotive attitude recognition model, extracts the coordinates of key points of the railway locomotive and self-propelled equipment in the image, calculates the vector difference between the vehicle body deflection angle and the wheel travel direction, and uses a convolutional neural network to perform semantic segmentation processing on each frame of the video frame sequence to identify the vehicle frame outline, bogie assembly, and wheel axle connection point location areas of the railway locomotive and self-propelled equipment. Within the vehicle frame outline area, the coordinates of the center points at both ends of the vehicle body are located using a key point regression algorithm, and the angle between the line connecting the two points and the horizontal baseline is calculated. The angle between the line connecting the two points and the horizontal baseline is defined as the vehicle body deflection angle. In some embodiments, the calculation of the vehicle body deflection angle uses the following formula:

[0084]

[0085] in: Indicates the vehicle body deflection angle. Indicates the coordinates of the center point of the front end of the vehicle body. This represents the coordinates of the center point at the rear of the vehicle. Within the area of ​​the wheel-axle connection point, the displacement of the center pixel of the same wheel in two adjacent frames is tracked. The direction of travel and displacement vector of the wheel per unit time are calculated. The direction of travel and displacement vector of the wheel per unit time are defined as the vector of the wheel's direction of travel. The vector corresponding to the vehicle body deflection angle is geometrically compared with the vector of the wheel's direction of travel. The calculated direction deviation value is used as the vector difference. It can be understood that the calculation of the vector difference involves vector subtraction and angle operation. The vehicle body deflection angle, the vector of the wheel's direction of travel, and the vector difference are packaged into an attitude feature vector. In specific implementation, for the railway shunting yard scenario, video frame sequences were used to capture side view images of the Dongfeng 4 diesel locomotive. The semantic segmentation model identified the vertex coordinates of the frame outline as (150, 300) and (450, 320) under sunny conditions, and (148, 298) and (448, 318) under cloudy low-light conditions. The calculation of the car body deflection angle showed that the change in coordinates caused the angle value to change from 5.71 degrees to 5.74 degrees. The data comparison listed the coordinate values ​​and angle calculation results under sunny and cloudy conditions.

[0086] The construction steps of the locomotive attitude recognition model include: collecting training video images containing different types of railway locomotives and self-propelled equipment under various working conditions; manually annotating the frame contours, bogie components, and wheel axle connection points in the training video images to generate a training dataset with location labels; constructing a semantic segmentation model backbone network based on a deep convolutional neural network and adding a feature pyramid structure to fuse multi-scale features; the semantic segmentation model backbone network is used to process training video images of different resolutions; two branch networks are connected in parallel at the output of the semantic segmentation model backbone network, where the first branch network outputs a car body contour segmentation map, and the second branch network outputs a heatmap of key points of the bogie and wheel axles; and using the training dataset to perform end-to-end analysis on the semantic segmentation model backbone network and the two branch networks. Joint training is performed, with the training loss function consisting of a weighted average of the cross-entropy loss of the vehicle body contour and the mean square error loss of the key point heatmap. After model training is completed, the network parameters are frozen, and the locomotive attitude recognition model is deployed to the attitude analysis module for real-time processing of video frame sequences. Optionally, random rotation and color jitter are used as data augmentation methods during training. Optionally, the key point heatmap is generated using a Gaussian kernel function to diffuse the labeled points. It can be understood that the feature pyramid structure helps to process locomotive images of different scales. In some embodiments, the training dataset contains video images of Shaoshan electric locomotives and Harmony diesel locomotives. Data comparison shows that the average number of labeled points on the Shaoshan electric locomotive frame contour is 12, and the average number of labeled points on the Harmony diesel locomotive frame contour is 10.

[0087] In one embodiment of the present invention, the behavior determination module inputs the vector difference between the vehicle body deflection angle and the wheel travel direction to the behavior compliance determination engine, and, in conjunction with preset work procedure constraints, identifies violation action segments that deviate from the standard working posture. (See also...) Figure 2 This process retrieves the operational procedure constraints that match the type of work currently being performed by the railway locomotive and self-propelled equipment. The operational procedure constraints define the maximum allowable deflection threshold and the tolerance range of the vector difference under specific working conditions. The deflection angle of the car body in the attitude feature vector is compared with the maximum deflection threshold. If it exceeds the threshold, it is marked as a deflection over-limit event. The vector difference in the attitude feature vector is compared with the tolerance range of the vector difference. If it falls outside the tolerance range, it is marked as a travel direction inconsistency event. Logical combination judgment is performed on the deflection over-limit events and travel direction inconsistency events in the continuous time series to identify composite violation states that simultaneously satisfy abnormal car body attitude and abnormal travel direction. The start and end timestamps of the identified single violation events or composite violation states are recorded to generate a time index of the violation action segment. In practical implementation, for scenarios involving shunting and coupling operations of railway locomotives within stations, the operational procedures stipulate that the maximum deflection threshold of the car body is 8 degrees, and the tolerance range for vector difference is [-3 degrees, +3 degrees]. When the attitude analysis module outputs a car body deflection angle of 9.5 degrees and a vector difference of -4.2 degrees, the behavior judgment module marks an event of deflection exceeding the limit and an event of inconsistent travel direction. The data comparison lists the specific values ​​under compliant and non-compliant states: under compliant state, the car body deflection angle is 7.2 degrees and the vector difference is 1.8 degrees; under non-compliant state, the car body deflection angle is 9.5 degrees and the vector difference is -4.2 degrees.

[0088] In its implementation, the behavioral compliance judgment engine is constructed by parsing and digitizing the standard operating procedure documents for railway locomotives and self-propelled equipment, extracting the vehicle body attitude constraints corresponding to different task types, including vehicle body deflection angle thresholds, wheel travel direction and vehicle body centerline angle thresholds, and threshold effective time windows. Based on the vehicle body attitude constraints, a rule knowledge base is constructed, indexed by task type, storing multiple sets of production rules consisting of conditions and conclusions. A Bayesian inference network is constructed, with nodes corresponding to conditions and conclusions in the production rules. The conditional probability distribution between nodes is calculated based on historical violation event data. It can be understood that the calculation of the conditional probability distribution is based on the frequency of different events occurring simultaneously in historical data. In some embodiments, the production rule for the "pushing vehicle group" task type stored in the rule knowledge base is: IF vehicle body deflection angle > 10 degrees AND vector difference < -5 degrees, THEN violation type is "pushing posture inaccuracy". It can be understood that the Bayesian inference network is used to handle probabilistic inference when measurement noise exists or some conditions are met. The posture feature vector output by the posture analysis module is used as real-time input, first undergoing deterministic rule matching in the rule knowledge base, then inputting into the Bayesian inference network for probabilistic risk inference. The matching results from the rule knowledge base and the inference results from the Bayesian inference network are integrated, outputting a judgment result containing violation type, confidence level, and time index. The judgment result is used by the behavior judgment module to generate the time index of the violation action segment. Optionally, the formula for calculating the violation probability output by the Bayesian inference network is expressed as:

[0089]

[0090] in: Indicates that evidence was observed Violation occurs under conditions such as specific pose feature vectors. The posterior probability, It represents the likelihood probability of this evidence appearing under circumstances of violation. It is the prior probability of the violation occurring. It is the marginal probability of evidence appearing.

[0091] In one embodiment of the present invention, see [reference] Figure 3The risk detection module acquires real-time monitoring footage from video acquisition terminals deployed on both sides of the railway crossing. It performs background modeling and foreground extraction on the footage, separating moving pedestrians, non-motorized vehicles, and motor vehicles. A deep learning target classification model is then used to classify and identify the extracted moving targets, distinguishing between motor vehicles, non-motorized vehicles, and pedestrians, and obtaining their position coordinates and trajectories within the footage. In some embodiments, for a four-lane level crossing scenario, the classification model identifies 3 pedestrian targets, 2 non-motorized vehicle targets, and 5 motor vehicle targets in a sunny afternoon video feed, outputting their pixel coordinates. Under low-light conditions in a cloudy evening, the same scene's classification model identifies 2 pedestrian targets, 1 non-motorized vehicle target, and 4 motor vehicle targets. Data comparison lists the number of targets and their position coordinate ranges identified under the two lighting conditions. Based on the physical width and clearance parameters of the railway crossing, [the module sets...]. The three-dimensional bounding box of the intrusion warning area extends to a safe distance outside the track centerline. It monitors the spatial relationship between the trajectory of a moving target and the three-dimensional bounding box in real time. Once an intersection between the target trajectory and the three-dimensional bounding box is detected, an intrusion alarm signal is immediately triggered. The intrusion alarm signal is correlated with the time index of the violation action segment to generate a comprehensive safety risk event record containing both personnel intrusion and equipment violations. In specific implementation, the coordinate sequence of a pedestrian's trajectory is [(100,200),(110,205),(120,210)], and the vertices of the projection area of ​​the three-dimensional bounding box in the image coordinate system are [(80,180),(200,180),(200,250),(80,250)]. When the trajectory point (120,210) falls within the projected polygon area, the system triggers the intrusion alarm signal.

[0092] The process of setting the three-dimensional spatial bounding box of the intrusion warning zone includes acquiring CAD design drawings or measured data of the railway crossing, analyzing the parameters of the inner rail spacing, ballast slope, and installation height of the crossing railings, calculating the effective passage width of the crossing based on the inner rail spacing, and then extending a preset safety redundancy distance outward to determine the horizontal cross-sectional dimensions of the three-dimensional spatial bounding box. In some embodiments, the formula for calculating the horizontal cross-sectional dimensions is:

[0093]

[0094] in: This represents the horizontal width of the three-dimensional bounding box. This indicates the effective passage width of the level crossing, calculated based on the inner spacing of the rails. The preset safety redundancy distance is used to calculate the vertical drop from the rail surface to the ground based on the ballast slope. Combined with the installation height of the level crossing railings, the vertical height range of the 3D bounding box is determined. It can be understood that the upper limit of the vertical height range is usually determined by the installation height of the level crossing railings. Combining the horizontal cross-sectional dimensions with the vertical height range, a trapezoidal cylindrical spatial model conforming to the actual terrain of the level crossing is constructed as the 3D bounding box. Optionally, with a ballast slope of 1:1.5, a vertical drop from the rail surface to the ground of 0.8 meters, and a level crossing railing installation height of 1.2 meters, the vertical height range of the 3D bounding box is set to 0 to 1.2 meters. The trapezoidal cylindrical spatial model is projected onto the video screen coordinate system to generate a 2D polygon detection area for real-time detection. It can be understood that the projection process involves camera calibration parameters and spatial coordinate transformation. In specific implementation, for two different railway level crossings, the inner rail spacing at level crossing A is 5 meters, and the safety redundancy distance is set to 1.5 meters. The horizontal width of the 3D bounding box is calculated. The rail spacing at level crossing B is 4.5 meters, with a safety margin of 1.2 meters. The horizontal width of the three-dimensional bounding box is calculated. The data comparison lists the calculation parameters and results for the two level crossings, which are 6.9 meters long. Optionally, the two-dimensional polygon detection area is formed by connecting the projection points of the bottom and top edges of the trapezoidal prism space model onto the image plane.

[0095] In one embodiment of the present invention, the cargo status monitoring module calls a dedicated cargo loading recognition model to detect the material accumulation outline inside the carriage or on the flatbed of the video screen where the self-propelled equipment is located, extracts the vertex coordinates of the material accumulation outline, and quantifies the current cargo loading rate by calculating the ratio of the outline area to the carriage floor area. At the same time, it detects whether there are obvious overhanging protrusions at the edges of the material accumulation outline. If so, it is marked as an off-center loading or overloaded state. The cargo loading rate is compared with a preset rated loading rate threshold. If it is lower than the lower limit or higher than the upper limit, a loading abnormality event is generated. The loading abnormality event is fused with the time index of the violation action segment to generate a comprehensive violation record including cargo status abnormality. In a specific implementation, the dedicated cargo loading recognition model detects a flatbed truck, outputting the vertex coordinate set of the material stacking contour as [(50,100),(150,100),(150,200),(50,200)], and the pixel vertex coordinates corresponding to the truck bed template as [(40,90),(160,90),(160,210),(40,210)], calculating a cargo loading rate of 64.5%. In another implementation, the model detected a significant protrusion at coordinates (155,205) on the material stacking contour of another truck, exceeding the boundary of the truck bed template. The system marked this detection as an overloading condition. Refer to Table 1 for a comparison of detection data for different loading conditions.

[0096] Table 1: Comparison of Detection Data for Loading Status

[0097]

[0098] The process of extracting the vertex coordinates of the material stacking contour and calculating the cargo loading rate includes edge smoothing of the material stacking contour to remove minor noise interference caused by lighting and shadows, using the convex hull algorithm to calculate the minimum bounding polygon of the processed contour, and obtaining the vertex coordinate set of the polygon. It can be understood that the convex hull algorithm can eliminate the concave parts inside the contour, obtaining a polygon representing the overall area occupied by the material. Based on the vertex coordinate set, the actual pixel area of ​​the material stacking contour is calculated using the shoelace formula, which is expressed as:

[0099]

[0100] in: Represents the actual pixel area of ​​the material accumulation outline. Indicates the number of vertices. Represents the first polygon The coordinates of each vertex are used. The pixel area of ​​the car floor is obtained from a pre-calibrated car floor template as a reference area. This template is obtained by calibrating the car floor in an empty state. The actual pixel area of ​​the material accumulation contour is divided by the pixel area of ​​the car floor; the resulting percentage is the cargo loading rate. Optionally, the lower limit of the rated loading rate threshold is set to 30%, and the upper limit is set to 80%. When the calculated cargo loading rate is 20.8%, a loading abnormality event is generated. In some embodiments, for the same car in the morning and evening, the calculated contour area is 25,500 pixels when the lighting is uniform in the morning, and 26,050 pixels when there is shadow in the evening. After edge smoothing, the contour area is corrected to 25,520 pixels. Data comparison shows that the smoothing process reduces the area error caused by lighting and shadows.

[0101] See Figure 4 This is a chart showing the continuous monitoring of loading rate changes for self-propelled railway equipment. It displays the monitoring results of cargo loading rate at eight time points throughout the day, primarily determining whether each time period falls within the normal loading rate range (30%–80%). Of the seven monitoring points, four time periods (08:00, 12:00, 14:00, and 18:00) exceeded the normal range, indicating significant fluctuations in equipment loading status. Loading rates below 30% at 08:00, 12:00, and 18:00 may indicate empty or lightly loaded operation. Loading rates exceeding 80% at 14:00 and 16:00 suggest risks of overloaded or unevenly loaded cargo. The loading rate spiked sharply to its peak between noon and afternoon (12:00–16:00) and then quickly declined, reflecting potential concentrated loading / unloading operations in the workflow. Overloaded periods (14:00–16:00) can easily cause safety hazards such as vehicle tilting and prolonged braking distance; while lightly loaded periods may result in waste of resources or unreasonable work plans.

[0102] In one embodiment of the present invention, the thermal imaging verification module simultaneously activates infrared thermal imaging video recording on the video acquisition terminals corresponding to high-heat-generating components of the railway locomotive's diesel engine compartment, traction motor, and braking system. It performs temperature threshold mapping processing on the infrared thermal imaging video stream, converting the grayscale image into a pseudo-color temperature heatmap. In the pseudo-color temperature heatmap, it automatically identifies hotspot areas with temperatures exceeding a preset warning threshold and records their temperature change trend curves. It can be understood that the preset warning threshold is set based on the materials and normal operating temperature ranges of different equipment components. The location information of the hotspot areas is spatially registered with the corresponding equipment component locations in the visible light video stream to identify the specific overheated component names. The temperature change trend curve is correlated with the time index of the violation action segment to generate an early warning analysis report containing the risk of mechanical overheating. In some embodiments, for the detection of the diesel engine compartment, the preset warning threshold is set to 150 degrees Celsius. A hot spot with a temperature of 165 degrees Celsius is identified in the infrared thermal imaging video stream. Its temperature rises from 155 degrees Celsius to 165 degrees Celsius within 10 consecutive seconds, forming an upward temperature change trend curve. At the same time, the attitude analysis module marks a vehicle body deflection exceeding the limit event. The thermal imaging verification module associates the hot spot with the time index of the deflection exceeding the limit event and generates a warning analysis report indicating that "the diesel engine component is overheating and accompanied by abnormal vehicle body attitude".

[0103] The process of spatially registering the location information of the heat-generating hotspot area with the corresponding device component location in the visible light video stream includes selecting a set of known key points of the device component as reference anchor points, manually or automatically calibrating the reference anchor points in the visible light video frame and the infrared thermal imaging video frame respectively, and calculating the affine transformation matrix between the visible light video frame and the infrared thermal imaging video frame based on the coordinates of the anchor points in the visible light video frame and the coordinates of the corresponding anchor points in the infrared thermal imaging video frame. In some embodiments, the reference anchor points used for registering the diesel engine cylinder head include the four center points of the cylinder head bolts, with coordinates of (201,350), (219,350), (201,368), and (219,368) in the visible light frame, and coordinates of the corresponding points in the infrared frame, which are (198,348), (216,348), (198,366), and (216,366) respectively. Using an affine transformation matrix, the coordinates of the hot spots in the infrared thermal imaging video frame are mapped to the coordinate system of the visible light video frame. In practice, the formula for calculating the affine transformation matrix is ​​expressed as follows:

[0104]

[0105] in: Represents the coordinates in an infrared thermal imaging video frame. This represents the corresponding coordinates after transformation to the visible light video frame coordinate system. This represents a 3x3 affine transformation matrix. In the visible light video frame, based on the mapped coordinates, the system looks up the corresponding device component label to determine the specific component to which the hot spot belongs. Optionally, the device component labels are pre-stored in a spatial lookup table, which maps specific polygonal regions in the visible light image to component names. The identified specific component name is bound to the corresponding temperature data and stored in the device health status database. The spatial lookup table is understood to be established by pre-labeling the devices in the visible light image. In a specific implementation, the center coordinates of a hot spot in the infrared frame are (210, 355). The affine transformation matrix M calculates its mapped coordinates in the visible light frame to be approximately (213, 357). A lookup in the spatial lookup table reveals that these coordinates are located within a polygonal region labeled "Cylinder liner of cylinder six." The system then identifies this hot spot as "Cylinder liner of cylinder six" overheating and binds the component name "Cylinder liner of cylinder six" with the temperature value of 165 degrees Celsius for storage. Optionally, for the detection of the traction motor, the preset warning threshold is 130 degrees Celsius, the infrared thermal imaging identifies the hot spot temperature as 145 degrees Celsius, and the coordinates are registered to correspond to the "non-drive end bearing" label area in the visible light image. The data comparison lists the warning thresholds and measured temperatures of different components: diesel engine cylinder liner area threshold 150 degrees Celsius, measured 165 degrees Celsius; traction motor bearing area threshold 130 degrees Celsius, measured 145 degrees Celsius.

[0106] See Figure 5 This is a graph showing the trend of event severity over time, illustrating the dynamic changes in event severity within a 9-second time window. During the escalation phase, the severity steadily climbs from an initial level 1, peaking at level 5 at the 7th second, indicating a continuous deterioration of the event and a gradual accumulation of risk. During the deterioration phase, the severity drops sharply from level 5 back to level 1, recovering to the initial level within 2 seconds, indicating that intervention measures were effective and the event was quickly controlled and mitigated. The high-risk window is concentrated between 5 and 7 seconds, representing a critical period requiring focused intervention. Examples include events such as overloading or abnormal vehicle posture, showing the entire process from occurrence to escalation and eventual resolution. The rapid decline in risk after 7 seconds indicates a timely and effective response. Close monitoring of the 3-7 second risk escalation period is crucial for triggering early warnings and preventing the situation from escalating to level 5.

[0107] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A safety control system for railway locomotives and self-propelled vehicles based on video AI, characterized in that, The system includes: The video acquisition module deploys multiple video acquisition terminals with zoom and low-light capabilities in key operating areas and mobile equipment operating zones along the target railway line to continuously acquire on-site operation video streams including railway locomotives and self-propelled equipment. The video processing module performs frame extraction and timestamp synchronization processing on the acquired on-site operation video stream to generate a video frame sequence associated with the operating status of the railway locomotive and self-propelled equipment. The attitude analysis module inputs the video frame sequence into a pre-trained locomotive attitude recognition model, extracts the coordinates of key points of the railway locomotive and self-propelled equipment in the image, and calculates the vector difference between the vehicle body deflection angle and the wheel travel direction. The behavior determination module inputs the vector difference between the vehicle body deflection angle and the wheel travel direction into the behavior compliance determination engine, and, in combination with the preset work procedure constraints, identifies the non-compliant action segments that deviate from the standard working posture. The evidence generation module, based on the timestamp of the occurrence of the violation action segment, traces back the original surveillance video corresponding to the video frame sequence, extracts and packages it to generate a violation event evidence data package, which includes context video segments before and after the violation and posture data logs.

2. The video AI-based railway locomotive and self-propelled vehicle safety control system according to claim 1, characterized in that, The video frame sequence is input into a pre-trained locomotive attitude recognition model to extract the coordinates of key points of the railway locomotive and its self-propelled equipment in the image, and to calculate the vector difference between the vehicle body deflection angle and the wheel travel direction, including: The convolutional neural network is used to perform semantic segmentation on each frame of the video frame sequence to identify the outline of the railway locomotive and self-propelled equipment, the bogie assembly, and the location area of ​​the wheel axle connection point; Within the frame contour area, the coordinates of the center points at both ends of the vehicle body are located using a key point regression algorithm, and the angle between the line connecting the two points and the horizontal baseline is calculated and defined as the vehicle body deflection angle. Within the location area of ​​the wheel axle connection point, the displacement of the center pixel of the same wheel in two adjacent frames is tracked, and the direction of travel and displacement vector of the wheel per unit time are calculated, which is defined as the vector of the direction of travel of the wheel. The vector corresponding to the vehicle body deflection angle is geometrically compared with the vector of the wheel travel direction, and the calculated direction deviation value is used as the vector difference. The vehicle body deflection angle, the vector of the wheel travel direction, and the vector difference are packaged into an attitude feature vector for subsequent compliance determination. The steps for constructing the locomotive attitude recognition model include: Training video images containing different types of railway locomotives and self-propelled equipment under various working conditions were collected. The chassis outline, bogie components and wheel axle connection points in the training video images were manually labeled to generate a training dataset with location labels. A semantic segmentation model backbone network based on a deep convolutional neural network was constructed, and a feature pyramid structure was added to fuse multi-scale features for processing training video images of different resolutions. Two branch networks are connected in parallel at the output of the backbone network of the semantic segmentation model. The first branch network outputs the vehicle body contour segmentation map, and the second branch network outputs the heat map of key points of the bogie and wheel axle. The semantic segmentation model backbone network and two branch networks are jointly trained end-to-end using the training dataset. The training loss function is a weighted sum of the cross-entropy loss of the vehicle body contour and the mean square error loss of the key point heatmap. After the model training is completed, the network parameters are frozen, and the locomotive attitude recognition model is deployed to the attitude analysis module for real-time processing of the video frame sequence.

3. The video AI-based railway locomotive and self-propelled vehicle safety control system according to claim 2, characterized in that, The vector difference between the vehicle body deflection angle and the wheel travel direction is input into the behavior compliance determination engine. Combined with preset work procedure constraints, it identifies violation action segments that deviate from the standard operating posture, including: Retrieve the operating procedure constraints that match the type of operation currently being performed by the railway locomotive and self-propelled equipment. The operating procedure constraints define the maximum allowable deflection threshold of the car body and the tolerance range of the vector difference under specific working conditions. The vehicle body deflection angle in the attitude feature vector is compared with the maximum deflection threshold of the vehicle body. If the threshold is exceeded, it is marked as a deflection over-limit event. The vector difference in the attitude feature vector is compared with the vector difference tolerance interval. If it falls outside the tolerance interval, it is marked as a travel direction inconsistency event. Logical combination judgment is performed on the deflection exceeding the limit event and the inconsistency of the direction of travel event in the continuous time series to identify the composite violation state that simultaneously satisfies the abnormal vehicle posture and the abnormal direction of travel. Record the start and end timestamps of the identified single violation event or complex violation status to generate a time index of the violation action segment.

4. The video AI-based railway locomotive and self-propelled vehicle safety control system according to claim 3, characterized in that, Also includes: The risk detection module acquires real-time monitoring images from video acquisition terminals deployed on both sides of railway crossings, performs background modeling and foreground extraction on the images, and separates moving pedestrians, non-motorized vehicles, and motor vehicles. The extracted moving targets are classified and identified using a deep learning target classification model, distinguishing between motor vehicles, non-motor vehicles and pedestrians, and obtaining their position coordinates and movement trajectories in the image. Based on the physical width and clearance parameters of the railway crossing, a three-dimensional spatial bounding box for the intrusion warning area is set, and the three-dimensional spatial bounding box extends to a safe distance outside the track centerline. The trajectory of the moving target and the spatial positional relationship between the three-dimensional bounding box are monitored in real time. Once the intersection of the target trajectory and the three-dimensional bounding box is detected, the intrusion alarm signal at the crossing is immediately triggered. The intrusion alarm signal at the crossing is associated with the time index of the violation action segment to generate a comprehensive safety risk event record that includes personnel intrusion and equipment violations.

5. The video AI-based railway locomotive and self-propelled vehicle safety control system according to claim 4, characterized in that, Based on the physical width and clearance parameters of the railway crossing, a three-dimensional spatial bounding box for the intrusion warning area is defined, including: Obtain the CAD design drawings or measured data of the railway crossing, and analyze the parameters of the inner rail spacing, ballast slope and the installation height of the crossing railing. The effective passage width of the crossing is calculated based on the inner spacing of the rails, and a preset safety redundancy distance is extended outward to determine the horizontal cross-sectional dimensions of the three-dimensional spatial bounding box. The vertical drop from the rail surface to the ground is calculated based on the slope of the ballast side slope, and the vertical height range of the three-dimensional spatial enclosure box is determined in combination with the installation height of the level crossing railing. By combining the horizontal cross-sectional dimensions with the vertical height range, a trapezoidal column space model that conforms to the actual terrain of the crossing is constructed, serving as the three-dimensional space bounding box. The trapezoidal cylindrical spatial model is projected onto the video frame coordinate system to generate a two-dimensional polygonal detection area for real-time detection.

6. The video AI-based railway locomotive and self-propelled vehicle safety control system according to claim 5, characterized in that, Also includes: The cargo status monitoring module uses a dedicated cargo loading recognition model to detect the material accumulation outline inside the carriage or on the flatbed of the self-propelled equipment in the video footage. Extract the vertex coordinates of the material accumulation contour, and quantify the current cargo loading rate by calculating the ratio of the contour area to the area of ​​the carriage floor. Simultaneously, detect whether there are obvious overhanging protrusions at the edge of the material accumulation contour. If so, mark it as an off-center loading or over-limit state. The cargo loading rate is compared with a preset rated loading rate threshold. If it is lower than the lower limit or higher than the upper limit, an abnormal loading event is generated. The loading anomaly event is fused with the time index of the violation action segment to generate a comprehensive violation record that includes the abnormal loading status.

7. The railway locomotive and self-propelled vehicle safety control system based on video AI according to claim 6, characterized in that, Extract the vertex coordinates of the material accumulation contour, and quantify the current cargo loading rate by calculating the ratio of the contour area to the area of ​​the carriage floor, including: The material accumulation contour is smoothed at the edges to remove minor noise interference caused by lighting and shadows; The minimum bounding polygon of the processed contour is calculated using the convex hull algorithm, and the set of vertex coordinates of the polygon is obtained. Based on the set of vertex coordinates, the actual pixel area of ​​the material accumulation contour is calculated using the shoelace formula. Obtain the pixel area of ​​the carriage floor from the pre-calibrated carriage floor template, and use it as the reference area; The percentage value obtained by dividing the actual pixel area of ​​the material stacking outline by the pixel area of ​​the carriage floor is the cargo loading rate.

8. The railway locomotive and self-propelled vehicle safety control system based on video AI according to claim 7, characterized in that, Also includes: The thermal imaging verification module simultaneously enables infrared thermal imaging video recording on the video acquisition terminals corresponding to the high-heat-generating components of the railway locomotive diesel engine compartment, traction motor, and braking system. The infrared thermal imaging video stream is subjected to temperature threshold mapping processing to convert the grayscale image into a pseudo-color temperature heatmap; In the pseudo-color temperature heat map, hot spots with temperatures higher than a preset warning threshold are automatically identified, and their temperature change trend curves are recorded. Spatially register the location information of the heat-generating hotspot area with the corresponding device component location in the visible light video stream to identify the specific name of the overheated component; The temperature change trend curve is correlated with the time index of the violation action segment to generate an early warning analysis report containing the risk of mechanical overheating.

9. The video AI-based railway locomotive and self-propelled vehicle safety control system according to claim 8, characterized in that, Spatially register the location information of the heat-generating hotspot area with the corresponding device component location in the visible light video stream to identify the specific name of the overheated component, including: A set of known key points of equipment components are selected as reference anchor points, and the reference anchor points are manually or automatically calibrated in visible light video frames and infrared thermal imaging video frames, respectively. The affine transformation matrix between the visible light video frame and the infrared thermal imaging video frame is calculated based on the coordinates of the anchor point in the visible light video frame and the coordinates of the corresponding anchor point in the infrared thermal imaging video frame. Using the affine transformation matrix, the coordinates of the heat-generating hotspot region in the infrared thermal imaging video frame are mapped to the coordinate system of the visible light video frame; In visible light video frames, based on the mapped coordinate positions, the device component labels corresponding to the coordinate positions are found, thereby determining the specific component to which the heat hotspot area belongs; The identified component names are linked to their corresponding temperature data and stored in the equipment health status database.

10. The video AI-based railway locomotive and self-propelled vehicle safety control system according to claim 9, characterized in that, The construction steps of the behavior compliance determination engine include: The standard operating procedure document for railway locomotives and self-propelled equipment is analyzed and digitized, and the car body attitude constraints corresponding to different types of operation tasks are extracted. The car body attitude constraints include the car body deflection angle threshold, the angle threshold between the wheel travel direction and the car body centerline, and the time window when the threshold takes effect. Based on the vehicle body attitude constraints, a rule knowledge base is constructed. The rule knowledge base is indexed by the task type and stores multiple sets of production rules consisting of conditions and conclusions. Construct a Bayesian inference network, wherein the nodes of the Bayesian inference network correspond to the conditions and conclusions in the production rules, and calculate the conditional probability distribution between nodes based on historical violation event data; The attitude feature vector output by the attitude analysis module is used as real-time input, first undergoes deterministic rule matching through the rule knowledge base, and then is input into the Bayesian inference network for probabilistic risk inference. The matching results from the rule knowledge base and the inference results from the Bayesian inference network are integrated to output a judgment result containing the violation type, confidence level, and time index. The judgment result is used by the behavior judgment module to generate the time index of the violation action segment.

Images