ST-GCN-based rail transit driver operation normalization monitoring method and device

Abstract

The invention relates to the technical field of event detection and behavior specification supervision, and discloses a rail transit driver operation specification monitoring method and device based on an improved ST-GCN model, and the method comprises the steps: determining various key specification operations involved in the start and stop process of an urban rail transit train driver, key joint points corresponding to the key standard operations are calibrated in advance; collecting video data of the driver executing the various key standard operations in the internal and external environments of the vehicle, and evaluating and marking the compliance of the driver operation behaviors in the video data; constructing an ST-GCN model based on dynamic topology and a domain prior fusion layer, and training the model by using the evaluated and labeled video data; and the trained ST-GCN model based on dynamic topology and field prior fusion is utilized to monitor driver operation in real time, and a prediction result is output. According to the device and the method, the real-time performance and accuracy of supervision can be improved, and safe operation of rail transit is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of event detection and behavior norm supervision, and in particular to a method and device for monitoring the operation norm of a rail transit driver based on ST-GCN. Background Art

[0002] In rail transit systems, supervision of train drivers’ operational compliance is crucial, because every operational decision made by the driver directly affects the safety and operational stability of the train. Therefore, ensuring that the driver strictly abides by the operational specifications, especially before driving, during driving, and after parking, is crucial to avoiding accidents and improving the safety of train operation.

[0003] In order to supervise the operation standardization of drivers, traditional methods mainly rely on manual supervision and regular inspections. However, manual supervision has certain limitations, it is difficult to achieve real-time and comprehensive monitoring, and it is easily affected by human factors. With the continuous expansion of rail transit systems and the increase in operating pressure, the effectiveness of manual supervision has gradually decreased, and it is difficult to meet the high requirements of modern rail transit for safety monitoring. In recent years, automated behavior monitoring methods based on image data have gradually been applied to other industries, such as worker safety monitoring at construction sites, providing a reference for the operation supervision of rail transit drivers. These methods usually use pose estimation technology, such as OpenPose, combined with spatiotemporal graph convolutional networks (ST-GCN) for behavior recognition and standardization detection. However, the traditional OpenPose+ST-GCN method still faces many challenges when applied to rail transit driver operation monitoring. Most of the existing monitoring methods are based on single-frame images for behavior recognition. Although objects and actions can be identified more accurately, it is difficult to capture the temporal changes of behavior. For example, the driver's inspection action before driving usually consists of multiple continuous actions. A single image cannot reflect the coherence of these actions, making it difficult to judge whether the operation is complete and standardized.

[0004] The posture estimation method based on a single moment is difficult to understand the context of the behavior, which affects the comprehensive evaluation of the driver's operation standardization. Complex operation processes require the system to have a deep understanding of the sequence and relationship of actions, and traditional methods perform poorly in this regard.

[0005] In addition, although sensor-based data collection methods, such as wearable sensors, can provide real-time physiological data and behavioral information, they also have limitations. The complexity of multi-sensor data fusion and the inconvenience to drivers make these methods difficult to be widely used in the operation supervision of rail transit drivers. Especially in the recognition of continuous behaviors, multiple instantaneous actions (such as checking whether the door is closed, looking down at the screen, etc.) are difficult to accurately judge without timing analysis.

[0006] Therefore, due to the above limitations, traditional methods are difficult to fully cope with the complex scenarios and diverse challenges in the supervision of rail transit driver operations. There is an urgent need for a monitoring solution that can more effectively capture continuous actions and timing information to improve the real-time and accuracy of supervision and ensure the safe operation of rail transit. Summary of the invention

[0007] In order to overcome the shortcomings of the prior art, the present invention provides a method and device for monitoring the operational compliance of rail transit drivers based on an improved ST-GCN model, so as to solve the technical problems that traditional methods are difficult to fully cope with the complex scenarios and diversified challenges in the operational supervision of rail transit drivers, and are difficult to achieve real-time, accurate and comprehensive monitoring and evaluation of the operational compliance of drivers, which brings risks to the safe operation of rail transit.

[0008] The purpose of the present invention can be achieved through the following technical solutions: In a first aspect, the present invention provides a method for monitoring the operation compliance of a rail transit driver based on an improved ST-GCN model, comprising: Determine the various key standard operations involved in the start-stop process of urban rail transit train drivers, and pre-calibrate the key joint points corresponding to each key standard operation; Collecting video data of the driver performing the multiple key standard operations in the vehicle and outside environments, and evaluating and marking the compliance of the driver's operating behaviors in the video data; Build an ST-GCN model based on dynamic topology and domain prior fusion layer and train the model using evaluated and annotated video data; The trained ST-GCN model based on dynamic topology and domain prior fusion is used to monitor the driver's operation in real time and output the prediction results.

[0009] Preferably, the multiple key standard operations are five, including: whether the doors are completely closed, whether there are objects or people stuck in the car, safety inspection of the environment inside and outside the car, inspection of the train braking system, and standardized confirmation of the train start operation.

[0010] Preferably, the step of constructing a ST-GCN model based on a dynamic topology and domain prior fusion layer and training the model using evaluated and annotated video data includes: Performing human posture estimation and skeleton data extraction on the evaluated and annotated video data, wherein the skeleton data includes key joint point coordinates and corresponding joint point confidences; According to the joint confidence and domain prior knowledge, the joint credibility weighting module is used to dynamically adjust the weight of each joint point to obtain a weighted skeleton sequence; The weighted skeleton sequence is input into the ST-GCN model for training, including: Performing domain adaptive skeleton normalization processing on the weighted skeleton sequence, adjusting the skeleton proportions based on the driver's body shape parameters, and obtaining a standardized skeleton sequence; The normalized standardized skeleton sequence is input into the ST-GCN model for feature extraction. In the feature extraction process of ST-GCN, the connectivity relationship of the joints is adaptively adjusted according to the current time sequence segment and the candidate operation type through a learnable dynamic topology mechanism. The domain prior fusion layer is embedded in the middle layer to fuse the normalized standardized skeleton sequence with the normative operation prior vector, guiding the features to be close to the standard operation mode in the field of rail transit. Among them, in the 9 convolution blocks of ST-GCN, the residual learning structure is retained after each convolution block, and the dynamic topology and domain prior fusion layer are inserted between some blocks to dynamically optimize feature extraction. Finally, the SoftMax function is used to calculate the probability that the input activity belongs to a specific standard operation, and the loss is calculated with the true value of the label to complete the model training.

[0011] Preferably, the performing human posture estimation and skeleton data extraction on the evaluated and annotated video data comprises: Using OpenCV to convert the evaluated and annotated video into an image sequence frame by frame and perform preprocessing; Input each frame of the preprocessed image into the pre-trained VGG-19-based convolutional neural network to extract the high-dimensional convolutional feature map of the image; The high-dimensional convolutional feature map is input into the multi-stage network of OpenPose to generate skeleton data of the human body in each frame of the image, wherein the skeleton data includes: coordinate information of each key joint point of the human body and a confidence score of the detection result of each key node, wherein the coordinate information of these key nodes includes but is not limited to the two-dimensional position coordinates of the shoulder, elbow, wrist, knee, and ankle.

[0012] Preferably, the weight of each joint point is dynamically adjusted using a joint credibility weighting module according to the joint point confidence and domain prior knowledge to obtain a weighted skeleton sequence, including: Get the key joint point coordinates and joint point confidence output by OpenPose; Dynamic weights are assigned based on the pre-calibrated key joints and domain prior importance, combined with the joint confidence; higher weights are assigned to joints that play a key role in the current standard operation and have high confidence, and lower weights are assigned to joints with low confidence or irrelevant; According to the key joint point coordinates and dynamically assigned weights, a weighted skeleton sequence is constructed and output.

[0013] Preferably, the multi-stage network of OpenPose adopts a multi-stage, two-branch structure, wherein branch one is used to estimate the probability distribution of the existence of key joints of the human body and generate a confidence map for each key joint point; branch two is used to calculate the component affinity field between joints to determine which joints belong to the same person and reconstruct the skeleton topology; through multiple iterative stages, OpenPose continuously optimizes the confidence map and the component affinity field, and finally outputs the coordinates of the key joint points of each frame image and the corresponding joint point confidence score.

[0014] Preferably, the dynamic topology mechanism specifically includes: Introducing a learnable graph attention layer before or in the spatiotemporal convolutional block for feature extraction in the spatiotemporal graph convolutional network; According to the current timing segment and the candidate operation type, the weights of the adjacency relationship between joints are adaptively calculated; higher weights are given to joint connections with high relevance in key operations, and joint connections that are not related to the operation are weakened.

[0015] Preferably, the method comprises: In the domain prior fusion layer, a prior vector corresponding to each normative operation is defined in advance based on expert knowledge; The spatiotemporal features extracted by the middle layer of ST-GCN are fused with the prior vector to make the features closer to the standard action patterns in the rail transit field.

[0016] Preferably, the method of using the trained ST-GCN model based on dynamic topology and domain prior fusion to monitor the driver's operation in real time and output the prediction result includes: The trained ST-GCN model outputs the operation probability corresponding to the driver's operation in real time; converting the operation probability into a canonical operation score; The standard operation score is compared with a preset safety threshold, and a warning prompt is issued when the score is lower than the threshold. The score is recorded and a personalized incentive strategy is formulated based on historical trends.

[0017] In a second aspect, the present invention further provides a device for monitoring the operation compliance of a rail transit driver based on an improved ST-GCN model, comprising: The operation definition unit is used to determine the various key standard operations involved in the start and stop process of the urban rail transit train driver, and pre-calibrate the key joint points corresponding to each key standard operation; A video acquisition unit, used to collect video data of the driver performing the multiple key standard operations in the vehicle and outside environments, and to evaluate and mark the compliance of the driver's operating behaviors in the video data; Model building unit, which builds the ST-GCN model based on the dynamic topology mechanism and domain prior fusion layer and trains the model using the evaluated and annotated video data; The result output unit is used to monitor the driver's operation in real time using the trained ST-GCN model based on dynamic semantic topology and domain prior fusion, and output the prediction results.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. The joint credibility weighting module SCWM of the present invention dynamically weights the OpenPose results and combines the spatiotemporal graph convolutional network (ST-GCN) based on dynamic topology to detect the operating behavior of rail transit train drivers in real time, which can accurately evaluate whether the driver complies with the operating specifications, thereby improving the intelligence and automation level of operation supervision; 2. The present invention monitors and outputs the probability of the driver's standard operation in real time, converts it into a standard operation score, and classifies it according to the set threshold, so as to issue warnings or rewards in time, ensure that the driver's operation meets the safety requirements, and reduce human errors and safety hazards; 3. The present invention utilizes domain prior information and personalized normalization strategies, which can not only accurately monitor action compliance, but also maintain high discrimination and universality among different drivers, ultimately improving the overall efficiency and level of rail transit safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 It is a flow chart of a method for monitoring the operation standardization of a rail transit driver based on an improved ST-GCN model provided by an embodiment of the present invention; Figure 2 It is a network architecture diagram of a rail transit driver operation normative monitoring method based on an improved ST-GCN model provided by an embodiment of the present invention; Figure 3 is the architecture diagram of OpenPose of the present invention; Figure 4 This is the ST-GCN architecture diagram based on dynamic topology and domain prior fusion of the present invention; Figure 5 It is a unit module diagram of a rail transit driver operation compliance monitoring device based on an improved ST-GCN model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0022] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0023] See also Figure 1-Figure 4 , an embodiment of the present invention provides a rail transit driver operation normative monitoring method 100 based on an improved ST-GCN model, comprising: S110: determining a variety of key standard operations involved in the start-stop process of the urban rail transit train driver, and pre-calibrating key joint points corresponding to each key standard operation; In this step S110, first, determine the standard operation actions and the importance of the joints. In this embodiment, by comprehensively considering the operating specifications of urban rail transit train drivers and the suggestions of traffic safety experts, determine a variety of common key standard operations taken by the driver during the start and stop of the train. Five common key standard operations are preferably taken, including: whether the door is completely closed, whether there are objects or people stuck in the car, safety inspection of the environment inside and outside the car, inspection of the train braking system, and standardized confirmation of the train start operation.

[0024] These five common key standard operations are the key links to ensure the safety, accuracy and efficiency of the driver's operation. The following are the specific operating specifications: 1. Before the train starts, check whether the doors are completely closed. The driver confirms whether all doors are completely closed and locked to avoid accidental door opening during the start-up process and ensure the safety of passengers.

[0025] 2. Confirm that there are no objects or people stuck in the car. Check whether there are objects or people stuck in the car and ensure that the doors are closed normally to avoid affecting the start-up due to stuck objects or people.

[0026] 3. Carry out a final safety check of the environment inside and outside the train. The driver checks whether there are any safety hazards inside and outside the train, ensures that there are no obstacles or other threats around the train, and ensures the safe start of the train.

[0027] 4. Check and ensure that the train's braking system is normal. The driver should check whether the train's braking system is working properly to ensure that it can stop quickly and safely when it needs to.

[0028] 5. Start the train according to standard operating procedures. The driver confirms all safety measures one by one according to the prescribed steps to ensure that the train complies with operating specifications when starting and avoid any safety loopholes.

[0029] When determining these five key operations, the expert group also pre-calibrated the key joints and action patterns involved in each operation. For example, for operations involving door inspections, the criticality of the arm and head joints can be pre-calibrated to provide a domain priori basis for subsequent joint credibility weighting. Specifically, the movements of the upper limbs (shoulder, elbow, wrist) and head (neck joint), eye direction, and hand posture are particularly important; the "environmental safety inspection" step pays more attention to the rotation of the driver's head and neck joints and the unique patterns generated by the upper limbs during touch or indication. This prior knowledge will provide a reference for later data processing stages (such as joint credibility weighting and dynamic topology construction).

[0030] In this step, the five operations are clearly associated with the importance of specific joint points so that they can be targeted in the subsequent weighted processing of skeleton data.

[0031] S120: collecting video data of the driver performing the multiple key standard operations in the vehicle interior and exterior environment, and evaluating and marking the compliance of the driver's operating behaviors in the video data; Step S120 includes two main steps: (1) Experimental environment construction and data collection Establish an experimental environment for stopping and starting a train when it arrives at a station, which is divided into two areas: inside and outside the car. Specifically, in the car area: install high-resolution cameras in front of, on the side, and on the top of the train driving platform, so that the driver's upper body, upper limbs, facial direction, and arm movements can be clearly captured; arrange auxiliary markers on the instrument panel, door controllers, and around the seats to facilitate the later determination of the correspondence between joint movements and specific control operations. Outside the car area: arrange a camera on the side, front, and rear of the train body to record the driver's movements of looking out to view the platform status or the environment outside the car from different angles. When the driver needs to turn his head, reach out to indicate, or use the observation window to check the environment, these cameras will provide supplementary perspectives for the subsequent OpenPose extraction of skeleton features, reducing the problem of single-view occlusion.

[0032] Train drivers were recruited to participate in the experiment, and participants were ensured to have three years or more of driving experience. Each driver first collected and registered basic physical data such as height and arm length before entering the experimental environment to support subsequent personalized adaptive normalization, specifically Domain Adaptive Skeleton Normalization (DASN).

[0033] During the experiment, each driver needs to perform operations in sequence according to the five operation specifications defined in step S110. To improve the diversity and robustness of the data: each driver conducts multiple rounds of experiments, and rests for three minutes after each round of experiments to reduce operational inaccuracies caused by fatigue; in some experiments, the lighting inside the car can be adjusted (simulating different platform environments), the camera position can be fine-tuned, or the layout of landmarks outside the car can be changed to test the adaptability of the model to environmental changes; each round of experiments is recorded in AVI format with a frame rate of 30fps and a resolution of 532×300. The video duration is about 20-30 seconds, covering the complete start-stop preparation process and standard operation steps.

[0034] (2) Evaluation and labeling of video data Rail transit safety experts evaluate and annotate the compliance of drivers’ operating behaviors based on recorded videos, providing high-quality supervision labels for model training.

[0035] Specifically, the experts reviewed the quality and compliance of each driver's movements in the five operations frame by frame, and gave a score and label (such as "fully compliant", "partially compliant", "non-compliant") to each standardized operation.

[0036] During the expert annotation process, the joint features related to the standard operation are further recorded, such as the joints most relevant to this step and their typical motion trajectories in the "door inspection" operation. These annotation data will provide additional guidance for the subsequent domain prior fusion layer (DPFL) and dynamic topology (DT) construction.

[0037] After the data is labeled, all data are quality controlled and statistically analyzed. The average video length of each subject (driver) is about 22.3 seconds, and each single operation is 3.5 seconds on average. This ensures that in subsequent training, ST-GCN can fully learn the spatiotemporal features of continuous skeletal sequences. If the driver's posture is too obscured or atypical movements make it difficult to identify key joints in the video, experts will add a "low confidence" label to the video to guide the joint credibility weighting module to dynamically reduce the impact of the joint point on feature modeling during training.

[0038] Through the above-mentioned refined data collection and labeling strategy, a high-quality, multi-perspective, and richly annotated information source is provided for subsequent methods (including joint credibility weighting, personalized skeleton normalization, dynamic topology construction, and domain prior fusion), enabling the model to more accurately and robustly identify the standardized operating behavior of rail transit drivers.

[0039] S130: Build a ST-GCN model based on dynamic topology and domain prior fusion layer and train the model using the evaluated and annotated video data; In step S130, the following two main steps are included: S131: OpenPose and skeleton data processing: First, the captured video clips are converted into images frame by frame using OpenCV and input into the OpenPose architecture (using the VGG-19 algorithm to extract features). Through multiple stages of convolution and part affinity field (PAF) estimation, OpenPose outputs the human joint positions and their confidence maps and joint connection information in each frame. In the present invention, the skeleton data output by OpenPose is not directly used for ST-GCN processing, but the "joint credibility weighting module" (SCWM) is introduced on this basis. This module dynamically adjusts the joint weights according to the OpenPose confidence of each joint point and the previously defined action domain prior. For the joint points required for key operations (such as the arm and head joints when checking the car door), if the confidence is high, its feature weight is increased; the influence on non-critical joint points or low confidence points is weakened, and the weighted skeleton sequence is obtained as the input of the downstream ST-GCN.

[0040] Specifically, the OpenPose of the present invention uses an architecture based on the VGG-19 algorithm, and its workflow is as follows: Figure 3 As shown, step S131 includes the following sub-steps: S1311: Performing human posture estimation and skeleton data extraction on the evaluated and annotated video data, wherein the skeleton data includes key joint point coordinates and corresponding joint point confidences, specifically including: S13111: Use OpenCV to convert the evaluated and annotated video into image sequences frame by frame and perform preprocessing; This step S13111 is used to implement video frame extraction and preprocessing. Specifically, before inputting into the OpenPose architecture, OpenCV is used to convert the collected raw video data (AVI format, 30fps, resolution 532×300) into image sequences frame by frame. To ensure the stability of pose estimation, the input image is slightly preprocessed, including resizing, brightness normalization, and noise suppression.

[0041] S13112: Input each preprocessed frame image into a pre-trained VGG-19-based convolutional neural network to extract a high-dimensional convolutional feature map of the image; This step S13112 is used to implement OpenPose model input and feature extraction: use the OpenPose architecture to estimate the human posture of each frame image. The front end of OpenPose uses a convolutional neural network based on VGG-19 to extract features from the input image and output a feature map F.

[0042] Assume the input image is , after the VGG-19 convolution and pooling layers, the feature map is obtained , where H, W are the height and width of the feature map, and C is the number of channels.

[0043] S13113: Input the high-dimensional convolutional feature map into the multi-stage network of OpenPose to generate skeleton data of the human body in each frame image, wherein the skeleton data includes: coordinate information of each key joint point of the human body and a confidence score of each key node detection result, wherein the coordinate information of these key nodes includes but is not limited to the two-dimensional position coordinates of the shoulder, elbow, wrist, knee, and ankle.

[0044] This step S13113 is used to implement multi-stage reasoning and part affinity field (PAF) estimation: Among them, OpenPose's multi-stage network adopts a multi-stage, two-branch structure, in which: Branch 1 (confidence map): Estimate the probability distribution of the existence of key joints of the human body and generate a confidence map for each joint point , where k=1,...,K, K is the number of joint points.

[0045] Branch 2 (PAFs): Calculate the affinity field between joints to determine which joints belong to the same character and reconstruct the skeleton topology. The output of PAFs is , where M is the number of bone connections, and each connection has 2 components representing the vector field.

[0046] Through multiple iterations, OpenPose continuously optimizes the confidence map and PAFs, and finally outputs the key joint coordinates in each frame and the corresponding confidence score:

[0047] in, is the coordinate position of joint k, is the confidence score of the joint.

[0048] S1312: According to the joint confidence and domain prior knowledge, the Skeletal Confidence Weighting Module (SCWM) is used to dynamically adjust the weight of each joint point to obtain a weighted skeleton sequence, including: After obtaining the joint point coordinates and confidences output by OpenPose, this embodiment further introduces a joint credibility weighting module to perform secondary processing on these original skeleton data. The design idea of ​​SCWM is to combine domain knowledge with the joint confidence given by OpenPose to adaptively weight the influence of each joint.

[0049] The process of dynamically adjusting the weights of each joint point using the joint credibility weighting module (SCWM) includes: S13121: Get the coordinates of key joint points and the confidence of joint points output by OpenPose; In step S13121, the coordinates of the key joint points and the confidence of the joint points output by OpenPose in step S1311 are obtained.

[0050] S13122: Calculate the joint credibility weight according to the pre-calibrated key joint points and domain prior importance; Specifically, step S13122 includes: (1) Domain prior and joint importance mapping: When determining the standard operations (the five key operations defined in step S110), the expert group has qualitatively marked the correlation between each operation and the joint. For example, for the "door closing check" operation, the upper arm and head joints have a greater weight; for the "environmental safety check" operation, the neck and torso joints are more critical. Define the domain importance vector ,in Indicates the importance of joint k in a specific standard operation scenario. This vector is obtained based on expert advice and operation process analysis.

[0051] (2) Joint credibility weight calculation: For each joint k, the OpenPose confidence Prior importance of domain Combined, calculate the joint credibility weight, that is, the weighted coefficient A feasible weighted formula is a linear combination plus regularization:

[0052] in, and It is an adjustable parameter used to balance the influence of confidence and domain prior on weight. The denominator is a normalization term to ensure that the sum of all weights is 1.

[0053] S13123: Based on the key joint point coordinates and the joint credibility weights, construct weighted skeletal features and dynamically adjust the joint credibility weights of the weighted skeletal features; wherein higher weights are assigned to joint points that play a key role in the current standard operation and have high confidence, and lower weights are assigned to joint points that have low confidence or are irrelevant.

[0054] Specifically, step S13123 includes: (1) Weighted skeleton data construction: The original joint coordinates Weighted processing is performed to obtain a weighted skeleton representation. The weighting is directly used for the joint point coordinates to construct the weighted skeleton features. ,in:

[0055] Considering that the joint point features are used as the node features of the graph in the subsequent ST-GCN, is the initial feature of joint point k. In the feature representation, the joint point coordinate information with higher weight is more influential in the subsequent convolution and graph structure processing.

[0056] (2) Low confidence exception handling: If some joint points Too low (such as less than a certain threshold ), they can be treated specially: If the joint is not critical in this operation ( If the value is lower, we can consider directly weakening the influence of this point, such as Forced lowering. If the joint is critical in operation ( If the confidence is low, you can try to compensate in subsequent steps through multi-frame temporal smoothing or multi-view fusion strategies.

[0057] S13124: Output weighted bone feature sequence.

[0058] After SCWM processing, each frame corresponds to a set of weighted skeleton feature sequences. These features have comprehensively considered the confidence of joint detection and domain prior knowledge, laying a more stable and targeted input foundation for the subsequent ST-GCN model to perform motion analysis at the spatiotemporal level.

[0059] Through the processing of step S131, the joint point information that originally relied solely on the output of OpenPose can be weighted and integrated based on domain knowledge and confidence information. In this way, in the subsequent spatiotemporal graph convolutional network processing, the model can focus more on the key joint points that really affect the recognition of standard operations, thereby improving the accuracy and stability of action recognition.

[0060] S132: ST-GCN model reasoning and training: The weighted skeleton feature sequence obtained after joint credibility weighting (SCWM) processing is input into the improved ST-GCN model for training and reasoning, such as Figure 4 This improvement introduces Domain Adaptive Skeletal Normalization (DASN), Dynamic Topology (DT) construction mechanism, and Domain Prior Fusion Layer (DPFL), thereby obtaining a spatiotemporal graph convolutional network (ST-GCN) model based on dynamic topology and domain prior fusion layer, thereby improving the accuracy and robustness of rail transit driver standard operation recognition.

[0061] After inputting the skeleton sequence corresponding to the continuous image frames, the skeleton is first proportionally adjusted according to the individual characteristics of the driver through the domain adaptive skeleton normalization (DASN) layer, so that the skeletal feature distribution of different drivers is more comparable. Then, the dynamic topology (DT) construction mechanism and the domain prior fusion layer (DPFL) are introduced in the feature extraction process of ST-GCN. DT adaptively adjusts the connection relationship of the joint points according to the candidate operation type of the current time sequence segment, thereby highlighting the joint group most relevant to the current operation; DPFL fuses the normalized skeletal features with the predefined normative operation prior vector to guide the features to be close to the standard operation mode in the rail transit field. In the 9 convolution blocks of ST-GCN, the residual learning structure is retained after each block, and DT and DPFL layers are inserted between some blocks to dynamically optimize feature extraction. Finally, the probability that the input activity belongs to a specific normative operation is calculated by the SoftMax function, and the loss is calculated with the true value of the label to complete the model training. Among them, (1) Domain Adaptive Skeleton Normalization (DASN): Input data and normalized target: weighted skeleton features obtained from step S13124 (K is the number of joint points), the continuous frame sequence constitutes the input sequence The height and arm length ratios of different drivers vary greatly, and directly inputting them into ST-GCN may cause the model to be affected by individual differences. In DASN, based on the pre-measured driver body parameters (such as height H driver , Wingspan L driver ) for scaling. A normalized mapping can be defined:

[0062] in, is the standardized skeleton sequence obtained after normalization. and It can be obtained based on the average joint position and scale statistics of the driver, and is used to normalize the skeleton coordinate distribution to a spatial range with similar scale at the individual level. Through DASN, the skeleton feature distribution of different drivers is more comparable, laying a unified benchmark for subsequent spatiotemporal feature extraction.

[0063] (2) Dynamic Topology (DT) Construction Mechanism Conventional ST-GCN uses a fixed backbone topology, i.e., a fixed adjacency matrix Characterize the connectivity between joints. However, in different standard operations, the role ratio of key joints and unimportant joints changes, and fixed topology is difficult to focus on specific operations.

[0064] DT adaptive adjacency matrix generation: In the embodiment of the present invention, a graph attention module is inserted before each spatiotemporal convolution block of ST-GCN to adaptively generate a dynamic adjacency matrix according to the characteristics of the current temporal segment. , we can use the dot-product attention method:

[0065] in, is the feature representation of the i-th joint point at time t, is the feature representation of the j-th joint point at time t (output from the previous layer or obtained after DASN processing), and is a trainable projection matrix. By calculating the correlation scores between different joint points and normalizing them, the dynamic adjacency matrix at time t is obtained For the joint point pair (i, j) that has a high correlation with the current candidate operation type, will be larger, thereby strengthening the influence of the edge; while for connections between irrelevant or unimportant joints, Assign a lower value to weaken its impact.

[0066] The dynamic topology (DT) mechanism can adaptively calculate the weight of the adjacency relationship between joints according to the current time sequence segment and the candidate operation type; it gives higher weights to the joint connections with high relevance in the key operation, and weakens the joint edges that are irrelevant to the operation. The dynamic topology (DT) is used to adaptively adjust the connection relationship between joints, strengthen the connectivity between key joints according to the normative requirements of the current operation, and the dynamic adjacency matrix generated by this operation is used to guide the subsequent training of skeletal features.

[0067] Domain Prior Fusion Layer (DPFL) Domain prior feature vector: In the previous expert annotation and statistical analysis, the corresponding prior vector has been defined for each standard operation , where d is the feature dimension. It reflects the typical skeletal motion characteristics, key joint point layout and timing pattern of the operation.

[0068] In addition, in order to align in dimension, the weighted skeleton coordinates obtained in the previous step need to be Upgrade to Matching feature space, first through the linear layer to map X to bone features , and then merge:

[0069] Feature fusion strategy: Insert the DPFL layer in the middle of ST-GCN to combine the currently extracted skeleton features (d is the number of hidden layer channels) and the operation prior To perform fusion. Linear mapping and weighted fusion can be used: ;in, is the expansion vector, Broadcast to K joint points; is a hyperparameter used to control the ratio of prior information fusion. If it is smaller, the prior has a greater impact on the current features, making the features closer to the standard operating mode in the rail transit field.

[0070] Therefore, in the domain prior fusion layer (DPFL), the prior vector corresponding to each standard operation is defined in advance based on expert knowledge; the spatiotemporal features (normalized skeleton features) extracted by the ST-GCN intermediate layer are fused with the prior vector to make the features closer to the standard action patterns in the rail transit field; the generalization uncertainty of pure data-driven methods is reduced, and the recognition accuracy and interpretability of specific standard operations are improved.

[0071] (4) Residual learning and 9 convolutional block architecture ST-GCN basic structure: The model contains 9 spatiotemporal convolution blocks, each of which includes spatiotemporal graph convolution and nonlinear activation function. and fusion features Input, the spatiotemporal graph convolution operation can be expressed as: ; where p traverses the spatial subset (such as the joint point itself, the subset of the neighboring nodes, and the subset of the distant neighboring nodes), is a trainable parameter, is the activation function (such as ReLU). Represents the block adjacency matrices assigned to different subset topologies.

[0072] Residual learning: Add residual connections between every two consecutive blocks to ensure the stability of deep network training and smooth gradient transfer: , the degradation problem caused by increased depth can be alleviated through residual learning.

[0073] (5) Loss calculation and model training Finally, during the training process, SoftMax classification is performed on each period, and the probability distribution of the period belonging to a specific standard operation is output. . Compare it with the true label In contrast, cross entropy loss is used for parameter update: ; Where t: represents the time step (Time Step), c represents the category index (Class). The model parameters are optimized through back propagation and gradient descent, including the normalization parameters in DASN, the graph attention weights in DT, the prior fusion ratio in DPFL, and the convolution kernel parameters of each layer of ST-GCN, so that the entire model can dynamically adapt to different drivers and different behavior scenarios, and improve the robustness and accuracy of standard operation recognition.

[0074] The above steps and formulas describe the innovative aspects of improving ST-GCN in dynamic topology construction, domain prior fusion, and domain adaptive normalization. This multi-level improvement enables the model to better capture the semantic and spatiotemporal characteristics of the rail transit driver's operation process, thereby providing more reliable and intelligent operation normative monitoring in practical applications.

[0075] S140: Use the trained ST-GCN model based on dynamic topology and domain prior fusion to monitor the driver's operation in real time and output the prediction results.

[0076] Specifically, step S140 includes: The trained ST-GCN model outputs the operation probability corresponding to the driver's operation in real time; converts the operation probability into a standard operation score; compares the standard operation score with a preset safety threshold, issues a warning when the score is lower than the threshold, records the score and formulates a personalized incentive strategy based on historical trends.

[0077] Specifically, during the driver's operation, the trained improved ST-GCN model is used to monitor the train driver in real time. The model outputs the probability that the operation belongs to a specific standard category in a certain time period, converts the probability into the driver's standard operation score and records it, and compares the score with the preset safety threshold; if the operation score is lower than the set safety threshold, the system immediately issues a real-time reminder, and feeds back to the driver through voice or display screen, prompting the driver that the operation he is performing is non-standard or has potential safety hazards, and provides corresponding improvement suggestions, such as "Please check whether the door is completely closed" or "Please confirm that the braking system is normal". This real-time feedback mechanism can help the driver adjust the operation in time and reduce safety hazards. In addition, the present invention uses the driver's score record to formulate a personalized incentive strategy to guide the driver to consciously improve his operating behavior, thereby improving the overall safety management level in long-term operation.

[0078] Specifically, the standardized operation scores recorded in real-time monitoring are analyzed and historical trend modeled, and personalized incentive strategies are formulated based on the driver's long-term performance. For example, positive incentives are provided to drivers whose scores continue to improve, and targeted training or safety reminders are given to drivers whose scores are frequently low. Through such long-term incentives and management strategies, the subjective initiative of drivers for standardized operations is enhanced, thereby reducing the probability of accidents overall and improving the safety level of rail transit operations.

[0079] In summary, a method for monitoring the standardization of rail transit driver operations based on an improved ST-GCN model in an embodiment of the present invention determines the standard operation as the experimental subject through the above steps 110-140, determines the key standard operations that the driver should take during the start and stop of the train, and builds an experimental environment to complete the data collection for training the model; uses the OpenPose architecture (based on the VGG-19 algorithm) to convert the collected video frame by frame into an image, and then extracts the human joint position, joint connection relationship and confidence in each frame, and combines the joint credibility weighting module (SCWM) to dynamically adjust the weight of each joint point to generate a weighted skeleton sequence, and then inputs these processed skeleton data into an improved model based on the spatiotemporal graph convolutional network (ST-GCN) to achieve classification and recognition of driver operations, such as Figure 2As shown in the figure; use the trained model to monitor the driver's operation in real time and output the probability of operating specifications, and give timely reminders when it is below the threshold; use the monitored driver's operating specification scores to formulate safety incentive policies, improve the driver's subjective initiative, and reduce the risk of accidents.

[0080] The above is an introduction to a method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment. Figure 5 A unit module diagram of a rail transit driver operation compliance monitoring device 200 based on an improved ST-GCN model according to an embodiment of the present invention is shown.

[0081] like Figure 5 As shown, the present invention provides a rail transit driver operation normative monitoring device 200 based on an improved ST-GCN model, comprising: The operation definition unit 201 is used to determine the various key standard operations involved in the start and stop process of the urban rail transit train driver, and pre-calibrate the key joint points corresponding to each key standard operation; The video acquisition unit 202 is used to collect video data of the driver performing the multiple key standard operations in the vehicle and outside environment, and evaluate and mark the compliance of the driver's operation behavior in the video data; A model building unit 203 builds an ST-GCN model based on a dynamic topology mechanism and a domain prior fusion layer and trains the model using the evaluated and annotated video data; The result output unit 204 is used to monitor the driver's operation in real time by using the trained ST-GCN model based on dynamic semantic topology and domain prior fusion, and output the prediction result.

[0082] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0083] The method of this embodiment and the above-mentioned reaction device are not described in detail here.

[0084] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the claims of the present invention.

Claims

1. A method for monitoring rail transit driver operation norms based on an improved ST-GCN model, characterized in that: include: Determine the various key standard operations involved in the start-stop process of urban rail transit train drivers, and pre-calibrate the key joint points corresponding to each key standard operation; Collecting video data of the driver performing the multiple key standard operations in the vehicle and outside environments, and evaluating and marking the compliance of the driver's operating behaviors in the video data; Build an ST-GCN model based on dynamic topology and domain prior fusion layer and train the model using evaluated and annotated video data; The trained ST-GCN model based on dynamic topology and domain prior fusion is used to monitor the driver's operation in real time and output the prediction results.

2. The method according to claim 1, characterized in that: There are five types of key standard operations, including: whether the doors are completely closed, whether there are objects or people stuck in the car, safety inspection of the environment inside and outside the car, inspection of the train braking system, and standardized confirmation of the train start operation.

3. The method according to claim 2, characterized in that: The ST-GCN model based on dynamic topology and domain prior fusion layer is constructed and the model is trained using the evaluated and annotated video data, including: Performing human posture estimation and skeleton data extraction on the evaluated and annotated video data, wherein the skeleton data includes key joint point coordinates and corresponding joint point confidences; According to the joint confidence and domain prior knowledge, the joint credibility weighting module is used to dynamically adjust the weight of each joint point to obtain a weighted skeleton sequence; The weighted skeleton sequence is input into the ST-GCN model for training, including: Performing domain adaptive skeleton normalization processing on the weighted skeleton sequence, adjusting the skeleton proportions based on the driver's body shape parameters, and obtaining a standardized skeleton sequence; The normalized standardized skeleton sequence is input into the ST-GCN model for feature extraction. In the feature extraction process of ST-GCN, the connectivity relationship of the joints is adaptively adjusted according to the current time sequence segment and the candidate operation type through a learnable dynamic topology mechanism. The domain prior fusion layer is embedded in the middle layer to fuse the normalized standardized skeleton sequence with the normative operation prior vector, guiding the features to be close to the standard operation mode in the field of rail transit. Among them, in the 9 convolution blocks of ST-GCN, the residual learning structure is retained after each convolution block, and the dynamic topology and domain prior fusion layer are inserted between some blocks to dynamically optimize feature extraction. Finally, the SoftMax function is used to calculate the probability that the input activity belongs to a specific standard operation, and the loss is calculated with the true value of the label to complete the model training.

4. The method according to claim 3, characterized in that: in, The performing of human posture estimation and skeleton data extraction on the evaluated and annotated video data comprises: Using OpenCV to convert the evaluated and annotated video into an image sequence frame by frame and perform preprocessing; Input each frame of the preprocessed image into the pre-trained VGG-19-based convolutional neural network to extract the high-dimensional convolutional feature map of the image; The high-dimensional convolutional feature map is input into the multi-stage network of OpenPose to generate skeleton data of the human body in each frame of the image, wherein the skeleton data includes: coordinate information of each key joint point of the human body and a confidence score of the detection result of each key node, wherein the coordinate information of these key nodes includes but is not limited to the two-dimensional position coordinates of the shoulder, elbow, wrist, knee, and ankle.

5. The method according to claim 4, characterized in that: The method dynamically adjusts the weight of each joint point using a joint credibility weighting module according to the joint point confidence and domain prior knowledge to obtain a weighted skeleton sequence, including: Get the key joint point coordinates and joint point confidence output by OpenPose; Calculate the joint credibility weight according to the pre-calibrated key joint points and domain prior importance; According to the key joint point coordinates and the joint credibility weights, a weighted skeleton feature is constructed and the weights of the joint credibility of the weighted skeleton feature are dynamically adjusted; wherein a higher weight is assigned to the joint points that play a key role in the current standard operation and have a high confidence, and a lower weight is assigned to the joint points that have a low confidence or are irrelevant; Output weighted bone feature sequence.

6. The method according to claim 5, characterized in that: The multi-stage network of OpenPose adopts a multi-stage, two-branch structure, wherein branch one is used to estimate the probability distribution of the existence of key joints of the human body and generate a confidence map for each key joint point; branch two is used to calculate the component affinity field between joints to determine which joints belong to the same person and reconstruct the skeleton topology; through multiple iterative stages, OpenPose continuously optimizes the confidence map and the component affinity field, and finally outputs the coordinates of the key joint points of each frame image and the corresponding joint point confidence score.

7. The method according to claim 6, characterized in that: The dynamic topology mechanism specifically includes: Introducing a learnable graph attention layer before or in the spatiotemporal convolutional block for feature extraction in the spatiotemporal graph convolutional network; According to the current timing segment and the candidate operation type, the weights of the adjacency relationship between joints are adaptively calculated; higher weights are given to joint connections with high relevance in key operations, and joint connections that are not related to the operation are weakened.

8. The method according to claim 6, characterized in that: The method comprises: In the domain prior fusion layer, a prior vector corresponding to each normative operation is defined in advance based on expert knowledge; The spatiotemporal features extracted by the middle layer of ST-GCN are fused with the prior vector to make the features closer to the standard action patterns in the rail transit field.

9. The method according to claim 8, characterized in that: The trained ST-GCN model based on dynamic topology and domain prior fusion is used to monitor the driver's operation in real time and output the prediction results, including: The trained ST-GCN model outputs the operation probability corresponding to the driver's operation in real time; converting the operation probability into a canonical operation score; The standard operation score is compared with a preset safety threshold, and a warning prompt is issued when the score is lower than the threshold. The score is recorded and a personalized incentive strategy is formulated based on historical trends.

10. A rail transit driver operation normative monitoring device based on an improved ST-GCN model, characterized in that: include: The operation definition unit is used to determine the various key standard operations involved in the start and stop process of the urban rail transit train driver, and pre-calibrate the key joint points corresponding to each key standard operation; A video acquisition unit, used to collect video data of the driver performing the multiple key standard operations in the vehicle and outside environments, and to evaluate and mark the compliance of the driver's operating behaviors in the video data; Model building unit, which builds the ST-GCN model based on the dynamic topology mechanism and domain prior fusion layer and trains the model using the evaluated and annotated video data; The result output unit is used to monitor the driver's operation in real time using the trained ST-GCN model based on dynamic semantic topology and domain prior fusion, and output the prediction results.

Images