A method for identifying abnormal behavior of subway drivers based on multimodal data fusion

Abstract

The present application provides a method for identifying abnormal behavior of subway drivers based on multimodal data fusion, the method comprising: identifying abnormal behavior of subway drivers based on multimodal data fusion, integrating track positioning and platform screen door sensor data to detect the train position and generate entry confirmation instructions; determining whether the driver has left his seat for inspection based on seat pressure distribution sensor data; generating exit completion events in combination with track circuit signals, ATO instructions, etc.; analyzing the touch pressure, voice commands, hand movements, etc. of the console to verify operational compliance, and triggering alarms in case of abnormalities; aggregating results at edge computing nodes and encrypting and transmitting them to the central dispatching system, dynamically mapping the driver's behavior in the digital twin cockpit, automatically triggering graded alarms and associating data snapshots in case of abnormalities; the present application can, with the help of multimodal data fusion and intelligent analysis, achieve accurate identification and real-time warning of subway driver behavior, significantly improving subway operation safety and emergency response efficiency.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a method for identifying abnormal behavior of subway drivers based on multimodal data fusion. Background Art

[0002] In subway operations, ensuring driving safety remains a core objective, and the standardization and timeliness of driver operations play a decisive role in achieving this goal. Traditional methods for monitoring subway driver behavior are relatively simple, relying primarily on manual observation or simple sensor detection. Manual observation is subject to significant subjective factors, making it difficult to achieve accurate, all-encompassing monitoring at all times, and unable to promptly capture complex and subtle abnormal behaviors. Simple sensor detection is typically limited to monitoring specific single behaviors or states. For example, seat pressure sensors alone can determine driver presence, or cameras can simply record the cab's image. These methods lack comprehensive analysis and in-depth mining of multi-dimensional information.

[0003] With the continuous expansion of subway operations, the increase in operating speed and the increasing complexity of lines, traditional monitoring methods can no longer meet the needs of accurately identifying and quickly responding to abnormal driver behavior, and cannot effectively prevent safety accidents caused by driver operating errors or violations. A more advanced and comprehensive monitoring technology is urgently needed to improve the safety of subway operations. Summary of the Invention

[0004] In view of this, an embodiment of the present application provides a method for identifying abnormal behavior of subway drivers based on multimodal data fusion, which can achieve accurate identification and real-time warning of subway driver behavior with the help of multimodal data fusion and intelligent analysis, thereby greatly improving the safety of subway operations and the efficiency of emergency response.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for identifying abnormal behavior of subway drivers based on multimodal data fusion, the method comprising:

[0007] Analyzing the door opening and closing sensor and the platform screen door status signals, generating a vehicle entry confirmation instruction when the door and the platform screen door are opened synchronously and the static background flag is valid; wherein the static background flag is triggered when the background motion vector is detected to be lower than a threshold value and lasts for a first specific time;

[0008] When the seat pressure distribution sensor detects that the pressure value is below the set threshold for more than a second specific time, the infrared thermal imaging camera and the key point extraction model are used to extract the driver's skeletal key points and calculate the deviation of the torso center of mass trajectory. If the lateral deviation of the center of mass exceeds a specific distance and lasts for a third specific time, it is determined that the driver has left the seat to perform a patrol operation;

[0009] Real-time monitoring of track circuit signal status. When receiving an ATO departure command from the Automatic Train Monitoring System (ATO), dynamic background analysis is initiated and door and platform screen door closing signals are simultaneously verified. When the motion vector continues to increase and the door status is closed, a vehicle departure completion event is generated.

[0010] Analyze the console touch pressure matrix data. When the interval between N consecutive pressure peaks is less than a fourth specified time, the standard console gesture is determined to be completed. The hand contour is captured by a multispectral camera, and the contact area and spatial angle between the hand and the console are calculated in real time through a 3D reconstruction model of the hand's key points. If the contact area is less than a specific percentage or the angle deviates from a specific degree for more than a fourth specified time, an abnormal alarm is triggered.

[0011] Aggregate entry / exit events, patrol status, and action recognition results at the edge computing node to generate structured logs. The data is encrypted and transmitted to the central dispatch system via the 5G dedicated network. The driver's behavior status is dynamically mapped in the digital twin cockpit. When an abnormality is detected in any link, a graded alarm is automatically triggered. The graded alarm includes at least a yellow warning for operation delay and a strong red warning for safety violations, and the corresponding video clips and sensor data snapshots are associated.

[0012] In a second aspect, an embodiment of the present application further provides a device for identifying abnormal behavior of subway drivers based on multimodal data fusion, the device comprising:

[0013] An entry confirmation module is configured to analyze signals from the door opening and closing sensors and the platform screen door status, and generate a vehicle entry confirmation instruction when the door and platform screen door are opened synchronously and a static background flag is valid; wherein the static background flag is triggered when a background motion vector is detected to be below a threshold value for a first specified period of time;

[0014] A patrol determination module is configured to, when the seat pressure distribution sensor detects that the pressure value has been below a set threshold for more than a second specified time, extract the driver's skeletal key points using an infrared thermal imaging camera and a key point extraction model, and calculate the deviation of the torso center of mass trajectory. If the lateral deviation of the center of mass exceeds a specified distance and persists for a third specified time, determine that the driver has left the seat to perform a patrol operation;

[0015] The exit confirmation module is used to monitor the status of track circuit signals in real time. When receiving the exit instruction from the automatic train monitoring system (ATO), it starts dynamic background analysis and simultaneously verifies the door and platform door closing signals. When the motion vector continues to increase and the door status is closed, a vehicle exit completion event is generated.

[0016] The abnormal alarm module is used to analyze the touch pressure matrix data of the console. When it detects that the interval between N consecutive pressure peaks is less than a fourth specified time, it determines that the standard gesture of the console is completed. The multispectral camera is used to capture the hand contour. The three-dimensional reconstruction model of the hand key points is used to calculate the contact area and spatial angle between the hand and the console in real time. If the contact area is less than a specific percentage or the angle deviates from a specific degree for more than a fourth specified time, an abnormal alarm is triggered.

[0017] The aggregation transmission module is used to aggregate entry / exit events, patrol status and action recognition results at the edge computing node, generate structured logs, and encrypt and transmit the data to the central dispatching system through the 5G dedicated network. The driver's behavior status is dynamically mapped in the digital twin cockpit. When an abnormality is detected in any link, a graded alarm is automatically triggered. The graded alarm includes at least a yellow warning for operation delay and a red strong warning for safety violation, and the corresponding video clips are associated with sensor data snapshots.

[0018] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to execute the method for identifying abnormal behavior of subway drivers based on multimodal data fusion as described in any one of the first aspects.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for identifying abnormal behavior of subway drivers based on multimodal data fusion as described in any one of the first aspects is executed.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] Through multimodal data fusion and intelligent analysis technology, real-time and accurate identification of train entry / exit status, driver inspections outside their seats, and operational compliance can be achieved. Combined with edge computing and 5G encrypted transmission, low-latency processing and secure transmission of behavioral data are ensured. Driver status is dynamically mapped in the digital twin cockpit and graded alarms are triggered (such as yellow warnings for operational delays and strong red warnings for safety violations), significantly improving the safety of subway operations and the efficiency of emergency response. At the same time, by correlating video clips with sensor data, the traceability of abnormal events is achieved, providing efficient and reliable technical support for the safety management and control of smart subways. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 10 is a flow chart of steps S101-S105 provided in an embodiment of the present application;

[0024] Figure 2 2 is a flow chart of steps S201-S203 provided in an embodiment of the present application;

[0025] Figure 3 3 is a flow chart of steps S301-S303 provided in an embodiment of the present application;

[0026] Figure 4 4 is a flow chart of steps S401-S403 provided in an embodiment of the present application;

[0027] Figure 5 Schematic diagram of the process of steps S501-S503 provided in an embodiment of the present application;

[0028] Figure 6 Schematic diagram of steps S601-S603 provided in an embodiment of the present application;

[0029] Figure 7 7 is a flow chart of steps S701-S703 provided in an embodiment of the present application;

[0030] Figure 8 Schematic diagram of the structure of a subway driver abnormal behavior identification device based on multimodal data fusion provided in an embodiment of the present application;

[0031] Figure 9 It is a schematic diagram of the composition structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0033] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein are for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0038] See also Figure 1 , Figure 1This is a flow chart of steps S101-S105 of the method for identifying abnormal behavior of subway drivers based on multimodal data fusion provided in the embodiment of the present application, which will be combined with Figure 1 Steps S101-S105 are shown for explanation.

[0039] In step S101, the door opening and closing sensor and the platform screen door status signal are analyzed, and when the door and the platform screen door are opened synchronously and the static background flag is valid, a vehicle entry confirmation instruction is generated; wherein, the static background flag is triggered when the background motion vector is detected to be lower than the threshold and lasts for a first specific time.

[0040] In step S102, when the seat pressure distribution sensor detects that the pressure value is lower than the set threshold for more than a second specific time, the driver's skeletal key points are extracted through the infrared thermal imaging camera and the key point extraction model, and the torso center of mass trajectory offset is calculated. If the lateral offset of the center of mass exceeds a specific distance and lasts for a third specific time, it is determined that the driver has left the seat to perform a patrol operation.

[0041] In step S103, the track circuit signal status is monitored in real time. When the automatic train monitoring system ATO exit instruction is received, dynamic background analysis is started and the door and platform door closing signals are verified synchronously. When the motion vector continues to grow and the door status is closed, a vehicle exit completion event is generated.

[0042] In step S104, the touch pressure matrix data of the console is analyzed. When it is detected that the interval between N consecutive pressure peaks is less than the fourth specific time, the standard gesture of the console is determined to be completed. The hand contour is captured based on the multispectral camera, and the contact area and spatial angle between the hand and the console are calculated in real time through the three-dimensional reconstruction model of the hand key points. If the contact area is less than a specific proportion or the angle deviates from the specific degree for more than the fourth specific time, an abnormal alarm is triggered.

[0043] In step S105, entry / exit events, patrol status and action recognition results are aggregated at the edge computing node to generate a structured log, and the data is encrypted and transmitted to the central dispatching system through the 5G dedicated network. The driver's behavior status is dynamically mapped in the digital twin cockpit. When an abnormality is detected in any link, a graded alarm is automatically triggered. The graded alarm includes at least a yellow warning for operation delay and a red strong warning for safety violation, and the corresponding video clips are associated with sensor data snapshots.

[0044] The above-mentioned embodiments of the present application are intended to monitor the driver's behavioral compliance in real time during key stages of train entry, exit, driving, and broadcasting operations, and identify the following three types of anomalies: operational delays (such as failure to confirm the exit signal in a timely manner), safety violations (such as leaving the seat without closing the door), and equipment misoperation (such as incorrect triggering of broadcasting instructions). Typical scenarios include the entry phase: train positioning, door / platform screen door status synchronization, and whether the driver is on duty. The exit phase: train start-up, door closure confirmation, and broadcasting compliance. The driving phase: the driver leaves the seat for inspection and gesture operation standardization.

[0045] The data sources for train status perception (entering / exiting the station) include the track positioning system and platform screen door sensors, integrating GPS / inertial navigation and Hall effect sensor data to eliminate positioning errors (such as relying on inertial navigation when GPS signals are lost in tunnels).

[0046] The door opening and closing sensor and the shield door status signal are synchronized in real time via the CAN bus, and the delay must be less than 50ms.

[0047] The following conditions must be met for entry confirmation: train positioning error <10cm + doors / platform screen doors open simultaneously + cab background is still (for 3 seconds).

[0048] The following conditions must be met for exiting the station: track circuit signal activation + sudden increase in background motion vector (detected by optical flow algorithm) + door closing (lasting 5 seconds).

[0049] To monitor the driver's on-duty status, this embodiment uses a seat pressure sensor to detect pressure (threshold 50 kg) and duration (>10 seconds), an infrared thermal imaging camera, and a YOLOv8-Pose model to extract key skeletal points and calculate the lateral displacement of the torso's center of mass (>15 cm and lasting for 5 seconds). The logical decision is: if the pressure value during an unseated inspection is less than the threshold and lasts for 10 seconds, the infrared camera is activated, the center of mass displacement exceeds the limit, and the driver is deemed unseated.

[0050] The data source for operation compliance testing is the console's touch pressure matrix, which measures the pressure peak interval (<0.5 seconds is considered continuous operation). A directional microphone array and lip movement recognition model verify the consistency between voice commands and lip movements (for example, "broadcast start" matches lip shape). A multispectral camera and a 3D hand reconstruction model calculate the contact area between the hand and the console (<30% is considered abnormal) and the spatial angle (>15° for 3 seconds). A compliant gesture is determined by three consecutive pressure peak intervals <0.5 seconds, with the lip movement matching the voice command. A strong red alert is triggered if the contact area is <30% or the angle is >15°.

[0051] For the classification rules of abnormality judgment and alarm classification, please refer to Table 1.

[0052]

[0053] Table 1

[0054] Edge computing nodes have the following functions: aggregating entry / exit events, patrol status, and motion recognition results, and generating structured logs (JSON format). Dynamic mapping of the digital twin cockpit can map driver behaviors (such as gestures and seat-leaving status) to the 3D virtual cockpit in real time with an accuracy of less than 2 cm. Visual alerts in this embodiment can highlight abnormal behaviors in red, with the ability to click to view the chain of evidence.

[0055] In some embodiments, see Figure 2 , Figure 2 2 is a flow chart of steps S201-S203 provided in an embodiment of the present application, and each step will be described in conjunction with the steps.

[0056] In step S201, a subway line feature database is constructed to record the length of each platform, curve curvature, and tunnel light intensity parameters;

[0057] In step S202, a multi-task learning network is designed based on the route feature database, wherein the main branch of the multi-task learning network is used to analyze the spatiotemporal continuity of the driver's gesture trajectory, and the auxiliary branch is used to dynamically adjust image enhancement parameters according to real-time tunnel lighting data;

[0058] In step S203, based on the curve curvature, braking distance parameters, current train speed and remaining platform length, the gesture recognition judgment time window is adaptively corrected, and when the train enters the tunnel, the action verification time is automatically extended to a specific multiple of the standard value; wherein, the specific multiple is greater than 1 times and less than or equal to 1.3 times.

[0059] Here, the construction of the subway line feature database includes:

[0060] (1) Data collection and storage.

[0061] Key parameters: Platform length: obtained through BIM model or laser scanning, accuracy ±5cm.

[0062] Curve curvature: measured by a track geometry inspection vehicle (such as GJ-6), the unit is 1 / m (for example, when the curvature radius R = 500m, the curvature = 1 / 500).

[0063] Tunnel light intensity: Deploy a light sensor (such as BH1750) with a sampling frequency of 1 Hz to record light values ​​(unit: lux) at different time periods (day / night) and tunnel depths.

[0064] Database design: Use a time series database (such as InfluxDB) to store illumination data, and a spatial database (such as PostGIS) to store station and track geometry data.

[0065] Data association: Bind light intensity to train position (based on track positioning system) through train ID and timestamp.

[0066] Regularly (e.g., monthly), patrol robots recollect curve curvature and platform length to correct for deviations caused by settlement or construction. Light intensity data is updated in real time, and a daytime illumination model (e.g., a Gaussian mixture model (GMM)) is established to predict future light changes.

[0067] The main branch of the multi-task learning network's network architecture is a spatiotemporal convolutional network (such as 3D-CNN + LSTM). The input is the console touch pressure matrix + gesture video frames (frame rate 30FPS), and the output is the spatiotemporal continuity score of the gesture trajectory (0-1). The score threshold is set to 0.8 (below the threshold is considered an abnormality).

[0068] The auxiliary branch's dynamic image enhancement module inputs real-time tunnel light intensity values ​​(queried from a route characteristics database) and outputs image enhancement parameters (such as gamma correction coefficients and contrast enhancement factors). The shared layer uses a ResNet-50 backbone network to extract low-level features and reduce computational redundancy.

[0069] The multi-task loss function is: ;

[0070] Among them, α=0.7, β=0.3 are weight coefficients, L gesture is the cross entropy loss for gesture classification, L enhancement It is the image quality loss after illumination enhancement (such as PSNR indicator).

[0071] The embodiment of the present application enhances the data in the following ways:

[0072] Simulating tunnel lighting changes: Randomly apply gamma correction (γ∈[0.5, 2.0]) to the training data. Adding noise: Add Gaussian noise (σ=0.05) to the pressure matrix data to improve model robustness.

[0073] The calculation formula for the dynamic correction of the judgment time window during the adaptive adjustment of gesture recognition parameters is:

[0074] Among them, T standard : Standard gesture verification duration (e.g. 2 seconds).

[0075] v: current speed of the train (m / s).

[0076] R: Curvature radius of the curve (m). For straight roads, take R=∞ (correction term is 0).

[0077] L remaining : Remaining length of the platform (m).

[0078] k: empirical coefficient (set to 0.1 based on simulation experiments) to ensure T adjusted ∈[1,1.3]T standard .

[0079] The trigger condition is that dynamic correction is automatically enabled when the train enters a curve (R<1000m) or a tunnel (light intensity<50lux).

[0080] For tunnel scenarios, the following optimizations can also be performed:

[0081] Image preprocessing: Dynamically adjust the infrared camera exposure time based on light intensity (e.g., when light intensity is <10 lux, the exposure time is extended to 1 / 30 second). After the auxiliary branch outputs the enhancement parameters, histogram equalization (CLAHE algorithm) is performed on the video frame.

[0082] Gesture verification strategy: Extend the verification time to 1.3 times the standard value (e.g., standard 2 seconds → 2.6 seconds in a tunnel). Lower the gesture similarity threshold (from 0.8 to 0.75) to avoid misjudgment due to image blur.

[0083] In some embodiments, see Figure 3 , Figure 3 This is a flow chart of steps S301-S303 provided in an embodiment of the present application, which will be explained in conjunction with each step.

[0084] In step S301, the ATO data stream is parsed in real time to obtain the train operation mode, emergency braking status and track occupancy information;

[0085] In step S302, based on the ATO data stream, when the train is in manual driving mode, a multispectral camera is activated to collect the driver's iris features, and the console grip sensor data is simultaneously extracted to construct an attention concentration scoring model, wherein the attention concentration scoring model is used to output an attention score;

[0086] In step S303, based on the track occupancy information and the attention score, when it is detected that no ATO ready signal is received after the train leaves the station and the driver's attention score is lower than the threshold, a composite alarm event integrating signal system abnormality and behavioral violation is triggered.

[0087] Here, the ATO (Automatic Train Operation) data stream determines the operating mode by reading a mode identifier via the CAN bus (e.g., 0 = automatic driving, 1 = manual driving). Emergency braking status is determined by parsing the brake command bit (1 = active, 0 = inactive). Track occupancy information, based on track circuit or axle counting system data, determines whether the current section is occupied (1 = occupied, 0 = free).

[0088] Use ROS (Robot Operating System) or DDS (Data Distribution Service) to achieve millisecond-level time alignment (error < 50ms) between ATO data, multispectral cameras, and grip force sensors, and build a finite state machine (FSM) to trigger different sensor strategies based on ATO mode switching (such as activating the iris camera during manual driving).

[0089] The attention concentration scoring model includes the following functions:

[0090] (1) Multimodal data acquisition.

[0091] Iris feature collection:

[0092] Equipment: Near-infrared multispectral camera (such as Hikvision DS-2CD3325), wavelength 850nm, frame rate 15FPS. Feature extraction: Use OpenCV's iris localization algorithm (such as the Daugman algorithm) to extract iris texture complexity (feature dimension = 128).

[0093] Grip force sensor data:

[0094] Equipment: Flexible pressure sensor array (e.g., FlexiForce A201), sampling rate 100 Hz. Feature extraction: Calculate grip force stability index (standard deviation σ, threshold σ < 0.5 N considered stable).

[0095] During the scoring model construction process, the model inputs iris texture complexity (128 dimensions) and grip stability (σ value). It outputs an attention score (0-100), with a threshold of 70 (a score below 70 indicates inattention). The algorithm can use a weighted fusion random forest model (W = 0.6 for iris features and W = 0.4 for grip stability), trained using historical data (sample size > 10,000).

[0096] In practical applications, the weight can be dynamically adjusted according to the train speed (for example, when the speed is > 60 km / h, the iris weight is increased to 0.7), or the model can be updated using incremental learning (online learning) to adapt to individual differences in drivers (for example, novice drivers have greater fluctuations in grip strength).

[0097] The trigger conditions for the compound alarm event trigger logic include:

[0098] Condition 1: The train does not receive the ATO ready signal after leaving the station (the train leaves the station as determined by track occupancy information, but the ATO mode has not been switched back to automatic driving).

[0099] Condition 2: Driver’s attention score < 70 (for 3 seconds).

[0100] If the ATO ready signal delay is >5 seconds and the track occupancy information shows that the section ahead is idle, it is determined to be a signal system abnormality (such as ZC communication failure).

[0101] Combined with the attention score, if the driver fails to stay focused during the signal anomaly, it will be judged as a composite violation (signal + behavior).

[0102] Alarm responses are graded, for example, yellow warning: only the signal system is abnormal (dispatch center prompts). Red strong warning: combined violations (triggering emergency braking + dispatch intervention).

[0103] The chain of evidence records the ATO data stream fragments (10 seconds before and after), iris video and grip sensor data for subsequent analysis.

[0104] In some embodiments, see Figure 4 , Figure 4 This is a flow chart of steps S401-S403 provided in an embodiment of the present application, which will be explained in conjunction with each step.

[0105] In step S401, a lightweight blockchain node is deployed on the train terminal to encrypt the driver's operation log, the hash value of the video key frame, and the signal system status code on the chain;

[0106] In step S402, based on the encrypted data link, the driver's biometric data is desensitized using differential privacy, and only the desensitized trajectory vector is uploaded after performing skeleton key point detection locally;

[0107] In step S403, the blockchain log is associated with the desensitized trajectory vector. When a hands-off-the-control-console timeout event is detected, the track vibration spectrum data and the carriage monitoring video clips for the corresponding period are automatically retrieved to generate a verifiable multimodal evidence chain.

[0108] Here, a lightweight blockchain node is deployed on the train's onboard terminal, equipped with optimized lightweight blockchain client software. A lightweight consensus algorithm (such as a simplified version of the Practical Byzantine Fault Tolerance (PBFT) algorithm) is used to ensure stable operation of the blockchain node without disrupting normal train operations. The blockchain node's network parameters are configured to enable secure communication with other blockchain nodes in the train operations management system. An encrypted communication channel is established to ensure the confidentiality and integrity of data transmission. During train operation, the onboard terminal collects real-time driver operation log data, including operation time, operation type (such as acceleration, deceleration, and gear shifting), and operation parameters. This operation log data is encrypted using a symmetric encryption algorithm (such as AES). The encryption key is generated and securely stored by the onboard terminal's security module. The encrypted operation log data is submitted to the blockchain network as transaction data through the blockchain node's interface and recorded in a blockchain block.

[0109] Onboard cameras capture real-time video data from within the train carriages and extract key frames at preset intervals. Each key frame is hashed to generate a unique hash value. This hash value is encrypted using an asymmetric encryption algorithm (such as RSA) and stored on-chain as transaction data. The train signaling system sends a status code to the onboard terminal in real time, indicating the train's operating status (e.g., normal operation, signal failure, etc.). The onboard terminal encrypts the received status code (using the same symmetric encryption algorithm as the driver's operation log) and then uploads the encrypted status code to the chain.

[0110] Differential privacy desensitization processing uses a differential privacy module deployed locally on the vehicle terminal. When collecting the driver's biometric data (such as facial images and hand gestures), the differential privacy module first pre-processes the biometric data to extract key features that characterize the driver's identity and behavior.

[0111] Skeletal keypoint detection and desensitized trajectory vector upload utilizes a pre-installed skeletal keypoint detection algorithm (such as a simplified version of the OpenPose algorithm) on the onboard terminal to detect the driver's gestures in real time, obtaining the coordinates of skeletal keypoints on the driver's hands, arms, and other parts. Based on the desensitized skeletal keypoint coordinates, the driver's gesture trajectory is calculated and represented as a desensitized trajectory vector. This desensitized trajectory vector only contains the driver's gesture movement trends and excludes sensitive information that could directly identify the driver. The onboard terminal uploads the desensitized trajectory vector via a secure communication channel to the train operations management system server for subsequent analysis and processing.

[0112] Data association is established on the train operations management system server, creating a blockchain log database and a desensitized trajectory vector database. Using timestamps as association fields, data stored on the blockchain, such as driver operation logs, video keyframe hash values, and signal system status codes, are associated with the desensitized trajectory vectors. For example, if an abnormality in the desensitized trajectory vector is detected at a certain time (such as hands leaving the control console), the blockchain log database can be searched for driver operation logs, video keyframe hash values, and signal system status code data around that time. Abnormal event detection and evidence chain generation establishes a timeout threshold for hands leaving the control console (e.g., 5 seconds). When the system detects that the driver's hands have been away from the control console for longer than this threshold, the abnormal event handling process is automatically triggered. Based on the associated data, the system automatically retrieves track vibration spectrum data (obtained from the train track monitoring system) and carriage surveillance video clips (retrieved from the video storage system based on video keyframe hash values) for the corresponding time period.

[0113] The generated multimodal chain of evidence is stored in a secure database and notified to relevant management personnel for processing. By reviewing the chain of evidence, managers can fully understand the train's operating status, driver operations, and surrounding environment at the time of the abnormal event, providing strong support for subsequent investigations and decision-making.

[0114] In some embodiments, see Figure 5 , Figure 5 This is a flow chart of steps S501-S503 provided in an embodiment of the present application, which will be explained in conjunction with each step.

[0115] In step S501, a holographic projection module is deployed in front of the driver's console, and a three-dimensional spatial mapping model of the operating interface is constructed through a ToF camera;

[0116] In step S502, based on the three-dimensional space mapping model, when a non-standard gesture is recognized, a dynamic AR guidance spot is generated. The color of the spot changes gradually according to the error type. When the operation is delayed, a yellow pulse is displayed, and when the trajectory deviates, a red spiral ripple is displayed.

[0117] In step S503, when a continuous operation error is detected, a vibration prompt signal synchronized with a standard operation frequency is generated in the corresponding function key area.

[0118] Here, edge blockchain nodes (such as a lightweight version of Hyperledger Fabric) are deployed on train terminals, storing only data relevant to the train itself, reducing storage and computing overhead. Data uploaded to the blockchain includes driver operation logs, including console keystrokes and handle movements, encrypted using a SHA-256 hash before being uploaded. Every 10 seconds, a keyframe is extracted (e.g., using OpenCV's frame difference method), its hash value is calculated, and uploaded to the blockchain. Signaling system status codes, such as ATO / ATP system status (e.g., mode switching, braking commands), are encrypted and uploaded to the blockchain with a timestamp.

[0119] A simplified version of Practical Byzantine Fault Tolerance (PBFT) is adopted, allowing only the onboard terminal and nodes in two adjacent carriages to participate in consensus, reducing communication delays (target consensus time < 200ms), and using the Merkle tree structure to store log hashes, reducing storage space (only the root hash needs to be stored for every 1,000 logs).

[0120] To desensitize biometric data, this embodiment uses MediaPipe or OpenPose on the vehicle terminal to extract 25 key points of the driver's upper body (such as shoulders, elbows, and wrists), generating a raw trajectory vector (dimension = 25 × 3 = 75, including x / y coordinates and confidence levels). Laplace noise (Laplace Mechanism) is added to each coordinate in the trajectory vector, with a privacy budget of ε = 0.5 (satisfying ε-differential privacy).

[0121] The multimodal evidence chain is generated in the following ways:

[0122] When the hands leave the console for a timeout, the velocity (v=Δd / Δt) and acceleration (a=Δv / Δt) of the wrist key points are calculated based on the desensitized trajectory vector. The trigger condition is that the hand velocity is <0.1m / s (stationary) for 3 consecutive seconds and the acceleration is close to 0 (no small movement).

[0123] The evidence chain association is based on blockchain log query. According to the timestamp of the abnormal event, the following information is retrieved from the blockchain within 10 seconds before and after: the driver's operation log hash value, the video key frame hash value (used to verify the integrity of the surveillance video), and the signal system status code (such as whether it is in manual driving mode).

[0124] Track vibration and video acquisition include track vibration spectrum data: Z-axis vibration data (sampling rate 1kHz) is obtained from on-board vibration sensors (such as the ADXL355 accelerometer), and the spectrum (0-100Hz) is calculated through FFT; and carriage surveillance video clips: 30 seconds of surveillance video (resolution 720p) before and after the abnormal period is extracted to verify whether the driver is talking to others or distracted.

[0125] To verify the chain of evidence, the embodiment of the present application compares the hash of the video keyframe stored in the blockchain with the hash value of the actual video clip to ensure that the data has not been tampered with. It uses a multimodal Transformer model to fuse the vibration spectrum, video clip and desensitized trajectory vector to generate an abnormal behavior confidence level (0-1).

[0126] In some embodiments, see Figure 6 , Figure 6 This is a flow chart of steps S601-S603 provided in an embodiment of the present application, which will be explained in conjunction with each step.

[0127] In step S601, a subway environmental parameter matrix is ​​established, wherein the subway environmental parameter matrix includes rail friction coefficient and pantograph contact network current data;

[0128] In step S602, an environment-aware adversarial network EAA-Net is constructed based on the environmental parameter matrix, and the sampling frequency of the skeleton key point tracking algorithm is increased when the rail is detected to be slippery;

[0129] In step S603, the rotation invariance parameters of the gesture recognition model are dynamically adjusted, and the weight attenuation coefficient of the fully connected layer of the action classification model is increased based on the rail friction coefficient.

[0130] Here, we first define and collect parameters. The data includes:

[0131] (1) Rail friction coefficient.

[0132] Sensor: Vehicle-mounted friction coefficient measuring instrument (such as Wheel-Rail Force Transducer), which calculates the μ value through the wheel-rail contact force and normal pressure.

[0133] Data classification: The μ value is divided into three levels: dry (μ>0.4), slippery (0.2<μ≤0.4), and icy (μ≤0.2).

[0134] (2) Pantograph contact network current: Sensor: Hall current sensor (such as LAH 50-P), measuring range 0-500A, sampling frequency 1kHz.

[0135] Feature extraction: Calculate the current fluctuation ratio (σ / μ, where σ is the standard deviation and μ is the mean) to evaluate the stability of the catenary.

[0136] The rail friction coefficient and contact network current are aligned by timestamp to construct a three-dimensional environmental parameter matrix. Anomaly detection is performed on the environmental parameter matrix based on the LSTM-Autoencoder. The threshold is set to trigger an alarm when the reconstruction error is greater than 0.15.

[0137] The Environment Aware Adversarial Network (EAA-Net) is based on an improved U-Net architecture, with dynamically adjustable convolution kernel depth. The skeletal keypoint tracking module is based on a lightweight version of AlphaPose and supports dynamic adjustment of the sampling frequency. The gesture recognition and action classification module is based on an improved 3D-ResNet network, supporting dynamic adjustment of rotation invariance parameters and weight decay coefficients.

[0138] When the rail friction coefficient μ≤0.4, the sampling frequency of the skeleton key point tracking algorithm is increased from 15Hz to 30Hz through the Dynamic Frame Rate Controller.

[0139] Curve curvature and gesture recognition are subject to rotational invariance parameter adjustment. The curve curvature κ (unit: 1 / m) is calculated using the vehicle's inertial measurement unit (IMU). When κ is greater than 0.01 (for sharp curves), the rotational invariance parameter θ of the gesture recognition model is expanded from ±15° to ±30°. This is achieved by inserting a learnable rotation layer after the convolutional layer of the 3D-ResNet. When the rail friction coefficient μ is ≤ 0.2 (for icing), the weight decay coefficient λ of the fully connected layer of the action classification model can be increased from 0.001 to 0.01. This can be achieved by dynamically modifying the L2 regularization term in an optimizer such as Adam.

[0140] In some embodiments, see Figure 7 , Figure 7 This is a flow chart of steps S701-S703 provided in an embodiment of the present application, which will be explained in conjunction with each step.

[0141] In step S701, a quantum key distribution module is integrated into the vehicle terminal to generate a true random encryption seed based on the track position;

[0142] In step S702, based on the quantum key, the post-quantum cryptography algorithm is used to encrypt the driver's operation video stream in blocks, and frame extraction analysis of key action frames is performed in the ciphertext domain;

[0143] In step S703, matching noise is superimposed during wireless transmission through a noise adaptive injection mechanism based on the encrypted seed and the tunnel electromagnetic interference data.

[0144] Here, a continuous-variable quantum key distribution (CV-QKD) scheme, based on the Gaussian modulated coherent state (GMCS) protocol, is employed to generate keys between the onboard terminal and the trackside quantum base station. Real-time data from track vibration sensors (such as the ADXL355 accelerometer) is used as a physical entropy source, combined with the randomness extraction algorithms of the QKD protocol (such as the Toeplitz hash) to generate a true random encryption seed. At a train speed of 80 km / h, the target key generation rate is ≥100 kbps, meeting the requirements for real-time encryption of video streams.

[0145] A new key is generated every 10 seconds, and a blockchain light node (deployed on the vehicle terminal) records the key version number and generation timestamp to ensure key traceability. Lattice-based cryptography-based key encapsulation mechanisms (such as the Kyber algorithm) are used to resist quantum computing attacks such as the Shor algorithm.

[0146] In summary, the embodiments of the present application have the following beneficial effects:

[0147] (1) Through the fusion of multimodal data (video, sensor, positioning, etc.) and dynamic analysis algorithms, accurate identification of abnormal behaviors such as driver leaving the seat, gesture operation, and distraction is achieved, effectively reducing the false alarm rate.

[0148] (2) Based on edge computing and quantum security encryption technology, it ensures real-time data processing and transmission anti-interference, and maintains high system availability in complex electromagnetic environments (such as tunnels).

[0149] (3) Automatically trigger graded alarms (such as yellow warning and red strong warning), associate video clips with sensor data, and minimize response delays, enabling dispatchers to intervene quickly.

[0150] (4) Quantum key distribution and post-quantum cryptographic algorithms are used to ensure data transmission security; blockchain and differential privacy technologies are combined to achieve the tamper-proof operation logs and biometric desensitization.

[0151] (5) Through the Environment Awareness Adversarial Network (EAA-Net), the image enhancement and gesture recognition parameters are dynamically adjusted to adapt to complex scenes such as tunnel dust and slippery rails, effectively improving the recognition accuracy.

[0152] (6) Covering the entire process of vehicle entry, driver inspection, and exit operation, combined with AR guidance and vibration prompts, the risk of human error is reduced and the efficiency of accident prevention is effectively improved.

[0153] Based on the same inventive concept, the embodiments of the present application also provide a subway driver abnormal behavior identification device based on multimodal data fusion corresponding to the subway driver abnormal behavior identification method based on multimodal data fusion in the first embodiment. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the above-mentioned subway driver abnormal behavior identification method based on multimodal data fusion, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0154] like Figure 8 As shown, Figure 8 : is a schematic diagram of the structure of a subway driver abnormal behavior identification device 800 based on multimodal data fusion provided in an embodiment of the present application. The subway driver abnormal behavior identification device 800 based on multimodal data fusion includes:

[0155] The station entry confirmation module 801 is used to analyze the vehicle door opening and closing sensor and the platform screen door status signal, and generate a vehicle station entry confirmation instruction when the vehicle door and platform screen door are opened synchronously and the static background flag is valid. The static background flag is triggered when the background motion vector is detected to be lower than a threshold value and lasts for a first specific time.

[0156] A patrol determination module 802 is configured to, when the seat pressure distribution sensor detects that the pressure value has been below a set threshold for more than a second specified time, extract the driver's skeletal key points using an infrared thermal imaging camera and a key point extraction model, and calculate the deviation of the torso center of mass trajectory. If the lateral deviation of the center of mass exceeds a specified distance and persists for a third specified time, determine that the driver has left the seat to perform a patrol operation;

[0157] The exit confirmation module 803 is used to monitor the status of track circuit signals in real time. When receiving the exit instruction from the automatic train monitoring system (ATO), it starts dynamic background analysis and simultaneously verifies the door and platform door closing signals. When the motion vector continues to increase and the door status is closed, a vehicle exit completion event is generated;

[0158] Abnormal alarm module 804 is used to analyze the console touch pressure matrix data. When it detects that the interval between N consecutive pressure peaks is less than a fourth specified time, it determines that the standard console gesture has been completed. The multispectral camera is used to capture the hand contour. The three-dimensional reconstruction model of the hand key points is used to calculate the contact area and spatial angle between the hand and the console in real time. If the contact area is less than a specified percentage or the angle deviates from a specified degree for more than a fourth specified time, an abnormal alarm is triggered.

[0159] The aggregation transmission module 805 is used to aggregate entry / exit events, patrol status and action recognition results at the edge computing node, generate structured logs, and encrypt and transmit the data to the central dispatch system through the 5G dedicated network. The driver's behavior status is dynamically mapped in the digital twin cockpit. When an abnormality is detected in any link, a graded alarm is automatically triggered. The graded alarm includes at least a yellow warning for operation delay and a red strong warning for safety violation, and the corresponding video clips are associated with sensor data snapshots.

[0160] Those skilled in the art should understand that Figure 8 The implementation functions of each unit in the device for identifying abnormal behavior of subway drivers based on multimodal data fusion 800 can be understood with reference to the relevant description of the method for identifying abnormal behavior of subway drivers based on multimodal data fusion. Figure 8 The functions of the various units in the device 800 for identifying abnormal behavior of subway drivers based on multimodal data fusion shown can be implemented by a program running on a processor, or by a specific logic circuit.

[0161] In one possible implementation, the method further includes:

[0162] Build a subway line feature database to record the length of each platform, curve curvature, and tunnel lighting intensity parameters;

[0163] Based on the route feature database, a multi-task learning network is designed, wherein the main branch of the multi-task learning network is used to analyze the spatiotemporal continuity of the driver's gesture trajectory, and the auxiliary branch is used to dynamically adjust image enhancement parameters according to real-time tunnel lighting data;

[0164] Based on the curve curvature, braking distance parameters, current train speed and remaining platform length, the gesture recognition judgment time window is adaptively corrected, and when the train enters a tunnel, the action verification time is automatically extended to a specific multiple of the standard value; wherein, the specific multiple is greater than 1 times and less than or equal to 1.3 times.

[0165] In one possible implementation, the method further includes:

[0166] Real-time analysis of ATO data streams to obtain train operation mode, emergency braking status, and track occupancy information;

[0167] Based on the ATO data stream, when the train is in manual driving mode, a multispectral camera is activated to collect the driver's iris features and simultaneously extract the console grip sensor data to construct an attention concentration scoring model, wherein the attention concentration scoring model is used to output an attention score;

[0168] Based on the track occupancy information and the attention score, when it is detected that no ATO ready signal is received after the train leaves the station and the driver's attention score is lower than a threshold, a composite alarm event integrating signal system abnormality and behavioral violation is triggered.

[0169] In one possible implementation, the method further includes:

[0170] Deploy lightweight blockchain nodes on train terminals to encrypt and upload driver operation logs, video key frame hash values, and signal system status codes to the blockchain;

[0171] Based on an encrypted data link, the driver's biometric data is desensitized using differential privacy. After performing skeletal key point detection locally, only the desensitized trajectory vector is uploaded.

[0172] The blockchain log is associated with the desensitized trajectory vector. When a hands-off-the-control-console timeout event is detected, the track vibration spectrum data and carriage surveillance video clips of the corresponding period are automatically retrieved to generate a verifiable multimodal evidence chain.

[0173] In one possible implementation, the method further includes:

[0174] A holographic projection module is deployed in front of the driver's console, and a three-dimensional spatial mapping model of the operating interface is constructed using a ToF camera;

[0175] Based on the three-dimensional spatial mapping model, when a non-standard gesture is recognized, a dynamic AR guidance light spot is generated. The color of the light spot changes gradually according to the error type. When the operation is delayed, a yellow pulse is displayed, and when the trajectory deviates, a red spiral ripple is displayed.

[0176] When continuous operation errors are detected, a vibration prompt signal synchronized with the standard operation frequency is generated in the corresponding function key area.

[0177] In one possible implementation, the method further includes:

[0178] Establish a subway environmental parameter matrix, including tunnel dust concentration, rail friction coefficient, and pantograph catenary current data;

[0179] Based on the environmental parameter matrix, an environmental awareness adversarial network (EAA-Net) is constructed to automatically enhance the convolution kernel depth of the image defogging module according to the real-time dust concentration. When wet rails are detected, the sampling frequency of the skeleton key point tracking algorithm is increased.

[0180] Based on the curve curvature data, the rotation invariance parameters of the gesture recognition model are dynamically adjusted, and the weight attenuation coefficient of the fully connected layer of the action classification model is increased based on the rail friction coefficient.

[0181] In one possible implementation, the method further includes:

[0182] Integrate a quantum key distribution module into the vehicle terminal to generate a true random encryption seed based on the track position;

[0183] Based on quantum keys, a post-quantum cryptographic algorithm is used to encrypt the driver's operation video stream in blocks, and frame extraction analysis of key action frames is performed in the ciphertext domain;

[0184] Based on the encrypted seed and tunnel electromagnetic interference data, matching noise is superimposed during the wireless transmission process through the noise adaptive injection mechanism.

[0185] The above-mentioned subway driver abnormal behavior recognition device based on multimodal data fusion has the following beneficial effects:

[0186] (1) Through the fusion of multimodal data (video, sensor, positioning, etc.) and dynamic analysis algorithms, accurate identification of abnormal behaviors such as driver leaving the seat, gesture operation, and distraction is achieved, effectively reducing the false alarm rate.

[0187] (2) Based on edge computing and quantum security encryption technology, it ensures real-time data processing and transmission anti-interference, and maintains high system availability in complex electromagnetic environments (such as tunnels).

[0188] (3) Automatically trigger graded alarms (such as yellow warning and red strong warning), associate video clips with sensor data, and minimize response delays, enabling dispatchers to intervene quickly.

[0189] (4) Quantum key distribution and post-quantum cryptographic algorithms are used to ensure data transmission security; blockchain and differential privacy technologies are combined to achieve the tamper-proof operation logs and biometric desensitization.

[0190] (5) Through the Environment Awareness Adversarial Network (EAA-Net), the image enhancement and gesture recognition parameters are dynamically adjusted to adapt to complex scenes such as tunnel dust and slippery rails, effectively improving the recognition accuracy.

[0191] (6) Covering the entire process of vehicle entry, driver inspection, and exit operation, combined with AR guidance and vibration prompts, the risk of human error is reduced and the efficiency of accident prevention is effectively improved.

[0192] like Figure 9 As shown, Figure 9 This is a schematic diagram of the structure of an electronic device 900 provided in an embodiment of the present application. The electronic device 900 includes:

[0193] A processor 901, a storage medium 902 and a bus 903, wherein the storage medium 902 stores machine-readable instructions executable by the processor 901. When the electronic device 900 is running, the processor 901 communicates with the storage medium 902 via the bus 903, and the processor 901 executes the machine-readable instructions to perform the steps of the method for identifying abnormal behavior of subway drivers based on multimodal data fusion described in the embodiment of the present application.

[0194] In actual application, the various components in the electronic device 900 are coupled together via the bus 903. It is understood that the bus 903 is used to realize the connection and communication between these components. In addition to the data bus, the bus 903 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 9 Various buses are labeled as bus 903.

[0195] The electronic device has the following beneficial effects:

[0196] (1) Through the fusion of multimodal data (video, sensor, positioning, etc.) and dynamic analysis algorithms, accurate identification of abnormal behaviors such as driver leaving the seat, gesture operation, and distraction is achieved, effectively reducing the false alarm rate.

[0197] (2) Based on edge computing and quantum security encryption technology, it ensures real-time data processing and transmission anti-interference, and maintains high system availability in complex electromagnetic environments (such as tunnels).

[0198] (3) Automatically trigger graded alarms (such as yellow warning and red strong warning), associate video clips with sensor data, and minimize response delays, enabling dispatchers to intervene quickly.

[0199] (4) Quantum key distribution and post-quantum cryptographic algorithms are used to ensure data transmission security; blockchain and differential privacy technologies are combined to achieve the tamper-proof operation logs and biometric desensitization.

[0200] (5) Through the Environment Awareness Adversarial Network (EAA-Net), the image enhancement and gesture recognition parameters are dynamically adjusted to adapt to complex scenes such as tunnel dust and slippery rails, effectively improving the recognition accuracy.

[0201] (6) Covering the entire process of vehicle entry, driver inspection, and exit operation, combined with AR guidance and vibration prompts, the risk of human error is reduced and the efficiency of accident prevention is effectively improved.

[0202] An embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When the executable instructions are executed by at least one processor 901, the method for identifying abnormal behavior of subway drivers based on multimodal data fusion described in the embodiment of the present application is implemented.

[0203] In some embodiments, the storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface storage, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0204] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0205] As an example, executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (for example, files storing one or more modules, subroutines, or code portions).

[0206] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0207] The computer-readable storage medium has the following advantages:

[0208] (1) Through the fusion of multimodal data (video, sensor, positioning, etc.) and dynamic analysis algorithms, accurate identification of abnormal behaviors such as driver leaving the seat, gesture operation, and distraction is achieved, effectively reducing the false alarm rate.

[0209] (2) Based on edge computing and quantum security encryption technology, it ensures real-time data processing and transmission anti-interference, and maintains high system availability in complex electromagnetic environments (such as tunnels).

[0210] (3) Automatically trigger graded alarms (such as yellow warning and red strong warning), associate video clips with sensor data, and minimize response delays, enabling dispatchers to intervene quickly.

[0211] (4) Quantum key distribution and post-quantum cryptographic algorithms are used to ensure data transmission security; blockchain and differential privacy technologies are combined to achieve the tamper-proof operation logs and biometric desensitization.

[0212] (5) Through the Environment Awareness Adversarial Network (EAA-Net), the image enhancement and gesture recognition parameters are dynamically adjusted to adapt to complex scenes such as tunnel dust and slippery rails, effectively improving the recognition accuracy.

[0213] (6) Covering the entire process of vehicle entry, driver inspection, and exit operation, combined with AR guidance and vibration prompts, the risk of human error is reduced and the efficiency of accident prevention is effectively improved.

[0214] In the several embodiments provided in this application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0215] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0216] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0217] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, platform server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0218] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for identifying abnormal behavior of subway drivers based on multimodal data fusion, characterized in that: The method comprises: Analyzing the door opening and closing sensor and the platform screen door status signals, generating a vehicle entry confirmation instruction when the door and the platform screen door are opened synchronously and the static background flag is valid; wherein the static background flag is triggered when the background motion vector is detected to be lower than a threshold value and lasts for a first specific time; When the seat pressure distribution sensor detects that the pressure value is below the set threshold for more than a second specific time, the infrared thermal imaging camera and the key point extraction model are used to extract the driver's skeletal key points and calculate the deviation of the torso center of mass trajectory. If the lateral deviation of the center of mass exceeds a specific distance and lasts for a third specific time, it is determined that the driver has left the seat to perform a patrol operation; Real-time monitoring of track circuit signal status. When receiving an ATO departure command from the Automatic Train Monitoring System (ATO), dynamic background analysis is initiated and door and platform screen door closing signals are simultaneously verified. When the motion vector continues to increase and the door status is closed, a vehicle departure completion event is generated. Analyze the console touch pressure matrix data. When the interval between N consecutive pressure peaks is less than a fourth specified time, the standard console gesture is determined to be completed. The hand contour is captured by a multispectral camera, and the contact area and spatial angle between the hand and the console are calculated in real time through a 3D reconstruction model of the hand's key points. If the contact area is less than a specific percentage or the angle deviates from a specific degree for more than a fourth specified time, an abnormal alarm is triggered. Aggregate entry / exit events, patrol status, and action recognition results at the edge computing node to generate structured logs. The data is encrypted and transmitted to the central dispatch system via the 5G dedicated network. The driver's behavior status is dynamically mapped in the digital twin cockpit. When an abnormality is detected in any link, a graded alarm is automatically triggered. The graded alarm includes at least a yellow warning for operation delay and a strong red warning for safety violations, and the corresponding video clips and sensor data snapshots are associated.

2. The method according to claim 1, characterized in that The method further comprises: Build a subway line feature database to record the length of each platform, curve curvature, and tunnel lighting intensity parameters; Based on the route feature database, a multi-task learning network is designed, wherein the main branch of the multi-task learning network is used to analyze the spatiotemporal continuity of the driver's gesture trajectory, and the auxiliary branch is used to dynamically adjust image enhancement parameters according to real-time tunnel lighting data; Based on the curve curvature, braking distance parameters, current train speed and remaining platform length, the gesture recognition judgment time window is adaptively corrected, and when the train enters a tunnel, the action verification time is automatically extended to a specific multiple of the standard value; wherein, the specific multiple is greater than 1 times and less than or equal to 1.3 times.

3. The method according to claim 1, characterized in that The method further comprises: Real-time analysis of ATO data streams to obtain train operation mode, emergency braking status, and track occupancy information; Based on the ATO data stream, when the train is in manual driving mode, a multispectral camera is activated to collect the driver's iris features and simultaneously extract the console grip sensor data to construct an attention concentration scoring model, wherein the attention concentration scoring model is used to output an attention score; Based on the track occupancy information and the attention score, when it is detected that no ATO ready signal is received after the train leaves the station and the driver's attention score is lower than a threshold, a composite alarm event integrating signal system abnormality and behavioral violation is triggered.

4. The method according to claim 1, wherein The method further comprises: Deploy lightweight blockchain nodes on train terminals to encrypt and upload driver operation logs, video key frame hash values, and signal system status codes to the blockchain; Based on an encrypted data link, the driver's biometric data is desensitized using differential privacy. After performing skeletal key point detection locally, only the desensitized trajectory vector is uploaded. The blockchain log is associated with the desensitized trajectory vector. When a hands-off-the-control-console timeout event is detected, the track vibration spectrum data and carriage surveillance video clips of the corresponding period are automatically retrieved to generate a verifiable multimodal evidence chain.

5. The method according to claim 1, characterized in that The method further comprises: A holographic projection module is deployed in front of the driver's console, and a three-dimensional spatial mapping model of the operating interface is constructed using a ToF camera; Based on the three-dimensional spatial mapping model, when a non-standard gesture is recognized, a dynamic AR guidance light spot is generated. The color of the light spot changes gradually according to the error type. When the operation is delayed, a yellow pulse is displayed, and when the trajectory deviates, a red spiral ripple is displayed. When continuous operation errors are detected, a vibration prompt signal synchronized with the standard operation frequency is generated in the corresponding function key area.

6. The method according to claim 1, characterized in that The method further comprises: Establishing a subway environmental parameter matrix, wherein the subway environmental parameter matrix includes rail friction coefficient and pantograph contact network current data; An environment-aware adversarial network (EAA-Net) is constructed based on the environmental parameter matrix to increase the sampling frequency of the skeleton key point tracking algorithm when slippery rails are detected. Based on the curve curvature data, the rotation invariance parameters of the gesture recognition model are dynamically adjusted, and the weight attenuation coefficient of the fully connected layer of the action classification model is increased based on the rail friction coefficient.

7. The method according to claim 1, characterized in that The method further comprises: Integrate a quantum key distribution module into the vehicle terminal to generate a true random encryption seed based on the track position; Based on quantum keys, a post-quantum cryptographic algorithm is used to encrypt the driver's operation video stream in blocks, and frame extraction analysis of key action frames is performed in the ciphertext domain; Based on the encrypted seed and tunnel electromagnetic interference data, matching noise is superimposed during the wireless transmission process through the noise adaptive injection mechanism.

8. A device for identifying abnormal behavior of subway drivers based on multimodal data fusion, characterized in that: The device comprises: An entry confirmation module is configured to analyze signals from the door opening and closing sensors and the platform screen door status, and generate a vehicle entry confirmation instruction when the door and platform screen door are opened synchronously and a static background flag is valid; wherein the static background flag is triggered when a background motion vector is detected to be below a threshold value for a first specified period of time; A patrol determination module is configured to, when the seat pressure distribution sensor detects that the pressure value has been below a set threshold for more than a second specified time, extract the driver's skeletal key points using an infrared thermal imaging camera and a key point extraction model, and calculate the deviation of the torso center of mass trajectory. If the lateral deviation of the center of mass exceeds a specified distance and persists for a third specified time, determine that the driver has left the seat to perform a patrol operation; The exit confirmation module is used to monitor the status of track circuit signals in real time. When receiving the exit instruction from the automatic train monitoring system (ATO), it starts dynamic background analysis and simultaneously verifies the door and platform door closing signals. When the motion vector continues to increase and the door status is closed, a vehicle exit completion event is generated. The abnormal alarm module is used to analyze the touch pressure matrix data of the console. When it detects that the interval between N consecutive pressure peaks is less than a fourth specified time, it determines that the standard gesture of the console is completed. The multispectral camera is used to capture the hand contour. The three-dimensional reconstruction model of the hand key points is used to calculate the contact area and spatial angle between the hand and the console in real time. If the contact area is less than a specific percentage or the angle deviates from a specific degree for more than a fourth specified time, an abnormal alarm is triggered. The aggregation transmission module is used to aggregate entry / exit events, patrol status and action recognition results at the edge computing node, generate structured logs, and encrypt and transmit the data to the central dispatching system through the 5G dedicated network. The driver's behavior status is dynamically mapped in the digital twin cockpit. When an abnormality is detected in any link, a graded alarm is automatically triggered. The graded alarm includes at least a yellow warning for operation delay and a red strong warning for safety violation, and the corresponding video clips are associated with sensor data snapshots.

9. An electronic device, characterized in that: include: A processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the method for identifying abnormal behavior of subway drivers based on multimodal data fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method for identifying abnormal behavior of subway drivers based on multimodal data fusion according to any one of claims 1 to 7 is executed.

Images