Subway driver abnormal behavior identification method based on multi-modal data fusion

Abstract

The invention provides a metro driver abnormal behavior recognition method based on multi-modal data fusion. The method comprises the steps that metro driver abnormal behaviors are recognized based on multi-modal data fusion, track positioning and platform screen door sensor data are fused to detect the position of a train, and a station entering confirmation instruction is generated; according to the data of the seat pressure distribution sensor, driver off-seat inspection is judged; generating an outbound completion event in combination with a track circuit signal, an ATO instruction and the like; analyzing console touch pressure, voice instructions, hand actions and the like to verify operation compliance, and triggering an alarm when the operation is abnormal; aggregating results at edge computing nodes, encrypting and transmitting the aggregated results to a central scheduling system, dynamically mapping driver behaviors in a digital twin cockpit, and automatically triggering hierarchical alarms and associating data snapshots when the driver behaviors are abnormal; according to the invention, accurate recognition and real-time early warning of subway driver behaviors can be realized by means of multi-modal data fusion and intelligent analysis, and the subway operation safety and the emergency response efficiency are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a method for identifying abnormal behavior of subway drivers based on multimodal data fusion. Background Art

[0002] In subway operations, ensuring driving safety remains a core objective, and the standardization and timeliness of driver operations play a decisive role in achieving this goal. Traditional methods for monitoring subway driver behavior are relatively simple, relying primarily on manual observation or simple sensor detection. Manual observation is subject to significant subjective factors, making it difficult to achieve accurate, all-encompassing monitoring at all times, and unable to promptly capture complex and subtle abnormal behaviors. Simple sensor detection is typically limited to monitoring specific single behaviors or states. For example, seat pressure sensors alone can determine driver presence, or cameras can simply record the cab's image. These methods lack comprehensive analysis and in-depth mining of multi-dimensional information.

[0003] With the continuous expansion of subway operations, the increase in operating speed and the increasing complexity of lines, traditional monitoring methods can no longer meet the needs of accurately identifying and quickly responding to abnormal driver behavior, and cannot effectively prevent safety accidents caused by driver operating errors or violations. A more advanced and comprehensive monitoring technology is urgently needed to improve the safety of subway operations. Summary of the Invention

[0004] In view of this, an embodiment of the present application provides a method for identifying abnormal behavior of subway drivers based on multimodal data fusion, which can achieve accurate identification and real-time warning of subway driver behavior with the help of multimodal data fusion and intelligent analysis, thereby greatly improving the safety of subway operations and the efficiency of emergency response.

[0005] The technical solution of the embodiment of the present application is implemented as follows: In a first aspect, an embodiment of the present application provides a method for identifying abnormal behavior of subway drivers based on multimodal data fusion, the method comprising: Analyzing the door opening and closing sensor and the platform screen door status signals, generating a vehicle entry confirmation instruction when the door and the platform screen door are opened synchronously and the static background flag is valid; wherein the static background flag is triggered when the background motion vector is detected to be lower than a threshold value and lasts for a first specific time; When the seat pressure distribution sensor detects that the pressure value is below the set threshold for more than a second specific time, the infrared thermal imaging camera and the key point extraction model are used to extract the driver's skeletal key points and calculate the deviation of the torso center of mass trajectory. If the lateral deviation of the center of mass exceeds a specific distance and lasts for a third specific time, it is determined that the driver has left the seat to perform a patrol operation; Monitor the signal status of the track circuit in real time. When receiving the ATO outbound instruction from the Automatic Train Supervision system, start dynamic background analysis and synchronously verify the door and platform screen door closing signals. When the motion vector continues to increase and the door status is closed, generate a vehicle outbound completion event; Analyze the touch pressure matrix data of the console. When it is detected that the interval between N consecutive pressure peaks is less than the fourth specific time, it is determined that the standard gesture on the console is completed. Based on the multi-spectral camera to capture the hand contour, the contact area and spatial angle between the hand and the console are calculated in real time through the 3D reconstruction model of hand key points. If the contact area is less than a specific proportion or the angle deviates from a specific degree for more than the fourth specific time, an abnormal alarm is triggered; Aggregate inbound / outbound events, patrol status, and action recognition results at the edge computing node, generate structured logs, and encrypt and transmit the data to the central dispatching system through the 5G private network. Dynamically map the driver's behavior status in the digital twin cockpit. When any abnormality is detected in any link, automatically trigger hierarchical alarms. The hierarchical alarms at least include a yellow warning for operation delay display, a red strong warning for safety violation display, and associate the corresponding video clips and sensor data snapshots.

[0006] In a second aspect, an embodiment of the present application further provides a subway driver abnormal behavior recognition device based on multi-modal data fusion. The device includes: An inbound confirmation module, configured to parse the door opening / closing sensor and platform screen door status signals, and generate a vehicle inbound confirmation instruction when the doors and platform screen doors are opened synchronously and the static background flag is valid; wherein, the static background flag is triggered when the detected background motion vector is lower than the threshold and lasts for the first specific time; A patrol determination module, configured to when the seat pressure distribution sensor detects that the pressure value is lower than the set threshold for more than the second specific time, extract the driver's bone key points through the infrared thermal imaging camera and the key point extraction model, and calculate the trunk centroid trajectory offset. If the centroid laterally deviates by more than a specific distance and lasts for the third specific time, it is determined that the driver leaves the seat to perform a patrol operation; An outbound confirmation module, configured to monitor the signal status of the track circuit in real time. When receiving the ATO outbound instruction from the Automatic Train Supervision system, start dynamic background analysis and synchronously verify the door and platform screen door closing signals. When the motion vector continues to increase and the door status is closed, generate a vehicle outbound completion event; An abnormal alarm module, configured to analyze the touch pressure matrix data of the console. When it is detected that the interval between N consecutive pressure peaks is less than the fourth specific time, it is determined that the standard gesture on the console is completed. Based on the multi-spectral camera to capture the hand contour, the contact area and spatial angle between the hand and the console are calculated in real time through the 3D reconstruction model of hand key points. If the contact area is less than a specific proportion or the angle deviates from a specific degree for more than the fourth specific time, an abnormal alarm is triggered; The aggregation and transmission module is used to aggregate inbound / outbound events, patrol status, and action recognition results at the edge computing node, generate structured logs, and encrypt and transmit the data to the central dispatching system through a 5G dedicated network, dynamically map the driver's behavior status in the digital twin cockpit, and automatically trigger hierarchical alarms when any abnormality is detected in any link. The hierarchical alarms at least include a yellow warning for operation delay display, a red strong warning for safety violation display, and associated corresponding video clips and sensor data snapshots.

[0007] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus. The processor executes the machine-readable instructions to perform the method for recognizing abnormal behaviors of subway drivers based on multimodal data fusion according to any one of the first aspects.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, it executes the method for recognizing abnormal behaviors of subway drivers based on multimodal data fusion according to any one of the first aspects.

[0009] The embodiments of the present application have the following beneficial effects: Through multimodal data fusion and intelligent analysis technologies, real-time and accurate recognition of the train's inbound / outbound status, the driver's leaving the seat for patrol, and operation compliance is achieved. Combining edge computing and 5G encrypted transmission ensures low-latency processing and secure transmission of behavior data, dynamically maps the driver's status in the digital twin cockpit and triggers hierarchical alarms (such as a yellow warning for operation delay and a red strong warning for safety violation), significantly improving the safety and emergency response efficiency of subway operations. At the same time, the traceability of abnormal events is realized through the association of video clips and sensor data, providing efficient and reliable technical support for the safety control of smart subways. Description of the Drawings

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0011] Figure 1 It is a flowchart of steps S101 - S105 provided by an embodiment of the present application; Figure 2 It is a flowchart of steps S201 - S203 provided by an embodiment of the present application; Figure 3 It is a schematic flowchart of steps S301 - S303 provided by an embodiment of the present application; Figure 4 It is a schematic flowchart of steps S401 - S403 provided by an embodiment of the present application; Figure 5 It is a schematic flowchart of steps S501 - S503 provided by an embodiment of the present application; Figure 6 It is a schematic flowchart of steps S601 - S603 provided by an embodiment of the present application; Figure 7 It is a schematic flowchart of steps S701 - S703 provided by an embodiment of the present application; Figure 8 It is a schematic structural diagram of a subway driver abnormal behavior recognition device based on multi - modal data fusion provided by an embodiment of the present application; Figure 9 It is a schematic composition structure diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the present application only serve the purposes of illustration and description, and are not used to limit the protection scope of the present application. Additionally, it should be understood that the schematic drawings are not drawn to actual scale. The flowcharts used in the present application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without logical context relationships may be reversed in order or implemented simultaneously. Furthermore, those skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0013] In the following descriptions, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0014] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application usually described and illustrated in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application claimed, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0015] In the following description, the terms "first / second / third" involved only distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0016] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the features stated thereafter, but does not exclude the addition of other features.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0018] See Figure 1 , Figure 1 is a schematic flowchart of steps S101 - S105 of the subway driver abnormal behavior recognition method based on multi-modal data fusion provided by the embodiments of the present application, and will be described in conjunction with Figure 1 the steps S101 - S105 shown.

[0019] In step S101, the door opening / closing sensor and the platform screen door status signal are parsed. When the doors and the platform screen doors are opened synchronously and the static background flag is valid, a vehicle arrival confirmation command is generated; wherein, the static background flag is triggered when the detected background motion vector is lower than the threshold and lasts for a first specific time.

[0020] In step S102, when the seat pressure distribution sensor detects that the pressure value is lower than the set threshold for a time exceeding a second specific time, the driver's skeletal key points are extracted through an infrared thermal imaging camera and a key point extraction model, and the trunk centroid trajectory offset is calculated. If the centroid laterally offsets by more than a specific distance and lasts for a third specific time, it is determined that the driver leaves the seat to perform a patrol operation.

[0021] In step S103, the signal state of the track circuit is monitored in real time. When receiving the ATO outbound instruction from the Automatic Train Supervision system, dynamic background analysis is started and the signal for the closed state of the car door and the platform screen door is verified synchronously. When the motion vector continues to increase and the car door state is closed, a vehicle outbound completion event is generated.

[0022] In step S104, the touch pressure matrix data of the console is analyzed. When it is detected that the interval between N consecutive pressure peaks is less than the fourth specific time, it is determined that the standard gesture of the operation console is completed. Based on the multi-spectral camera to capture the hand contour, the contact area and the spatial angle between the hand and the operation console are calculated in real time through the three-dimensional reconstruction model of hand key points. If the contact area is less than a specific proportion or the angle deviates from a specific degree for more than the fourth specific time, an abnormal alarm is triggered.

[0023] In step S105, the inbound / outbound events, the inspection status and the action recognition results are aggregated at the edge computing node to generate a structured log, and the data is encrypted and transmitted to the central dispatching system through the 5G private network, and the driver behavior status is dynamically mapped in the digital twin cockpit. When any abnormality is detected in any link, a hierarchical alarm is automatically triggered. The hierarchical alarm includes at least a yellow warning for operation delay display, a red strong warning for safety violation display and the association of the corresponding video clip and the sensor data snapshot.

[0024] The above embodiments of the present application aim to monitor the behavior compliance of the driver in key links such as train inbound, outbound, driving and broadcasting operations in real time, and identify the following three types of abnormalities: operation delay (such as failure to confirm the outbound signal in time), safety violation (such as leaving the seat without closing the car door), and equipment misoperation (such as wrongly triggering the broadcast instruction). Typical scenarios include the inbound stage: train positioning, synchronization of the car door / platform screen door status, and whether the driver is on duty. The outbound stage: train start, confirmation of car door closing, and compliance of broadcasting. The driving stage: the driver leaves the seat for inspection and the standardization of gesture operations.

[0025] The data sources for train state perception (inbound / outbound) include the track positioning system and the platform screen door sensor, which fuse GPS / inertial navigation and Hall effect sensor data to eliminate positioning errors (such as relying on inertial navigation when the GPS signal is lost in the tunnel).

[0026] The car door opening / closing sensor and the platform screen door state signal synchronize the car door and the platform screen door states in real time through the CAN bus, and the delay needs to be less than 50ms.

[0027] The inbound confirmation needs to meet the following conditions: train positioning error < 10cm + synchronous opening of the car door / platform screen door + stillness of the cab background (lasting for 3 seconds).

[0028] The outbound completion needs to meet the following conditions: activation of the track circuit signal + sudden increase in the background motion vector (detected by the optical flow algorithm) + car door closed (lasting for 5 seconds).

[0029] For the monitoring of the driver's on-duty status, the data sources adopted in the embodiments of this application are seat pressure sensors, which detect the pressure value (threshold: 50 kg) and the duration (> 10 seconds), and an infrared thermal imaging camera + YOLOv8-Pose model, which extracts the key points of the driver's skeleton and calculates the lateral offset of the trunk centroid (> 15 cm and lasting for 5 seconds). The logical determination is as follows: when the pressure value < the threshold and lasts for 10 seconds during the off-seat inspection tour → activate the infrared camera → the centroid offset exceeds the limit → determine that the driver is off the seat.

[0030] The data sources for the detection of operation compliance are the console touch pressure matrix, which detects the pressure peak interval (< 0.5 seconds is regarded as continuous operation), a directional microphone array + a lip movement recognition model, which verifies the consistency between the voice command and the lip movement (such as the lip shape matching for "Broadcast start"), and a multispectral camera + a hand three-dimensional reconstruction model, which calculates the contact area between the hand and the operation console (< 30% is regarded as abnormal) and the spatial angle (> 15° and lasting for 3 seconds). When the gesture is compliant, it is that the pressure peak interval is < 0.5 seconds for 3 consecutive times → the lip movement matches the voice command, and when the contact area < 30% or the angle > 15°, a strong red warning is triggered.

[0031] Please refer to Table 1 for the classification rules of abnormal determination and alarm classification.

[0032]

[0033] Table 1 The edge computing node has the following functions: aggregating in / out event, inspection status, and action recognition results to generate a structured log (in JSON format). The digital twin cockpit dynamic mapping can map the driver's behavior (such as gestures, off-seat status) to the 3D virtual cockpit in real time, with an error < 2 cm. The visual alarm in the embodiments of this application can be that abnormal behaviors are highlighted in red and support clicking to view the evidence chain.

[0034] In some embodiments, refer to Figure 2 , Figure 2 which is the flow schematic diagram of steps S201 - S203 provided by the embodiments of this application and will be described in combination with each step.

[0035] In step S201, a subway line feature database is constructed to record the length of each platform, the curvature of the bend, and the tunnel light intensity parameters; In step S202, based on the line feature database, a multi-task learning network is designed, where the main branch of the multi-task learning network is used to analyze the spatio-temporal continuity of the driver's gesture trajectory, and the auxiliary branch is used to dynamically adjust the image enhancement parameters according to the real-time tunnel light data; In step S203, based on the bend curvature, braking distance parameter, current train speed, and remaining platform length, adaptively correct the gesture recognition decision time window, and automatically extend the action verification duration to a specific multiple of the standard value when the train enters the tunnel; where the specific multiple is greater than 1 times and less than or equal to 1.3 times.

[0036] Here, the construction of the subway line feature database includes: (1) Data collection and storage.

[0037] Key parameters: Platform length: Obtained through BIM model or laser scanning, with an accuracy of ±5 cm.

[0038] Bend curvature: Measured based on a track geometry inspection vehicle (such as GJ-6 type), with the unit of 1 / m (for example, when the curvature radius R = 500 m, the curvature = 1 / 500).

[0039] Tunnel light intensity: Deploy light sensors (such as BH1750), with a sampling frequency of 1 Hz, and record the light values (unit: lux) at different time periods (day / night) and tunnel depths.

[0040] Database design: Use a time series database (such as InfluxDB) to store light data, and a spatial database (such as PostGIS) to store platform and track geometry data.

[0041] Data association: Bind the light intensity to the train position (based on the track positioning system) through the train ID and timestamp.

[0042] Regularly (such as monthly), re-collect the bend curvature and platform length through inspection robots to correct the deviations caused by settlement or construction. The light intensity data is updated in real time, and a day-night light model (such as Gaussian Mixture Model GMM) is established to predict future light changes.

[0043] The main branch of the network architecture of the multi-task learning network is a spatio-temporal convolutional network (such as 3D-CNN + LSTM), with the input being the console touch pressure matrix + gesture video frames (frame rate 30 FPS), and the output being the spatio-temporal continuity score (0-1) of the gesture trajectory. The score threshold is set to 0.8 (a value lower than the threshold is determined as abnormal).

[0044] The input of the dynamic image enhancement module of the auxiliary branch is the real-time value of the tunnel light intensity (queried from the line feature database), and the output is the image enhancement parameters (such as Gamma correction coefficient, contrast enhancement factor). The shared layer uses a ResNet-50 backbone network to extract low-level features and reduce computational redundancy.

[0045] The multi-task loss function is: ; Among them, α = 0.7, β = 0.3 are weight coefficients, Lgesture is the cross entropy loss for gesture classification, L enhancement It is the image quality loss after illumination enhancement (such as PSNR indicator).

[0046] The embodiment of the present application enhances the data in the following ways: Simulating tunnel lighting changes: Randomly apply gamma correction (γ∈[0.5, 2.0]) to the training data. Adding noise: Add Gaussian noise (σ=0.05) to the pressure matrix data to improve model robustness.

[0047] The calculation formula for the dynamic correction of the judgment time window during the adaptive adjustment of gesture recognition parameters is:

[0048] Among them, T standard : Standard gesture verification duration (e.g. 2 seconds).

[0049] v: current speed of the train (m / s).

[0050] R: Curvature radius of the curve (m). For straight roads, take R=∞ (correction term is 0).

[0051] L remaining : Remaining length of the platform (m).

[0052] k: empirical coefficient (set to 0.1 based on simulation experiments) to ensure T adjusted ∈[1,1.3]T standard .

[0053] The trigger condition is that dynamic correction is automatically enabled when the train enters a curve (R<1000m) or a tunnel (light intensity<50lux).

[0054] For tunnel scenarios, the following optimizations can also be performed: Image preprocessing: Dynamically adjust the infrared camera exposure time based on light intensity (e.g., when light intensity is <10 lux, the exposure time is extended to 1 / 30 second). After the auxiliary branch outputs the enhancement parameters, histogram equalization (CLAHE algorithm) is performed on the video frame.

[0055] Gesture verification strategy: Extend the verification time to 1.3 times the standard value (e.g., standard 2 seconds → 2.6 seconds in a tunnel). Lower the gesture similarity threshold (from 0.8 to 0.75) to avoid misjudgment due to image blur.

[0056] In some embodiments, see Figure 3 , Figure 3 This is a flow chart of steps S301-S303 provided in an embodiment of the present application, which will be explained in conjunction with each step.

[0057] In step S301, the data stream of ATO is parsed in real time to obtain the train operation mode, emergency braking state, and track occupancy information; In step S302, based on the data stream of the ATO, when the train is in the manual driving mode, a multi-spectral camera is activated to collect the driver's iris features, and the data of the grip force sensor on the console is synchronously extracted to construct an attention concentration scoring model, where the attention concentration scoring model is used to output an attention score; In step S303, based on the track occupancy information and the attention score, when it is detected that the ATO ready signal has not been received after the train departs and the driver's attention score is lower than the threshold, a composite alarm event that combines signal system anomalies and behavior violations is triggered.

[0058] Here, the operation mode of the ATO (Automatic Train Operation) data stream is to read the mode identifier through the CAN bus (e.g., 0 = automatic driving, 1 = manual driving). The emergency braking state is obtained by parsing the braking instruction bit (1 = activated, 0 = not activated). The track occupancy information is based on the track circuit or axle counter system data to determine whether the current section is occupied (1 = occupied, 0 = idle).

[0059] Use ROS (Robot Operating System) or DDS (Data Distribution Service) to achieve millisecond-level time alignment (error < 50ms) of ATO data with the multi-spectral camera and the grip force sensor and construct a finite state machine (FSM), and trigger different sensor strategies according to the ATO mode switch (e.g., activate the iris camera during manual driving).

[0060] The attention concentration scoring model includes the following functions: (1) Multi-modal data acquisition.

[0061] Iris feature acquisition: Device: Near-infrared multi-spectral camera (such as Hikvision DS-2CD3325), wavelength 850nm, frame rate 15FPS. Feature extraction: Use the iris localization algorithm of OpenCV (such as Daugman algorithm) to extract the iris texture complexity (feature dimension = 128).

[0062] Grip force sensor data: Device: Flexible pressure sensor array (such as FlexiForce A201), sampling rate 100Hz. Feature extraction: Calculate the grip force stability index (standard deviation σ, a threshold of σ < 0.5N is considered stable).

[0063] During the construction of the scoring model, the model inputs the iris texture complexity (128 dimensions) + the grip strength stability (σ value). The output is the attention score (0 - 100), and the threshold is set at 70 (a score below 70 is determined as inattentiveness). The algorithm can adopt a weighted fusion random forest model (W = 0.6 for iris features, W = 0.4 for grip strength stability), and is trained with historical data (sample size > 10,000).

[0064] In practical applications, the weights can be dynamically adjusted according to the train speed (e.g., when the speed > 60 km / h, the iris weight is increased to 0.7), or the model can be updated using incremental learning (Online Learning) to adapt to individual differences of drivers (e.g., novice drivers have greater grip strength fluctuations).

[0065] The triggering conditions for the composite alarm event trigger logic include: Condition 1: The ATO ready signal is not received after the train departs (it is determined that the train has left the station through track occupancy information, but the ATO mode has not switched back to automatic driving).

[0066] Condition 2: The driver's attention score < 70 (for 3 consecutive seconds).

[0067] If the delay of the ATO ready signal > 5 seconds and the track occupancy information shows that the section ahead is idle, it is determined that there is an abnormality in the signal system (such as a ZC communication failure).

[0068] Combined with the attention score, if the driver fails to maintain focus during the signal abnormality, it is determined as a composite violation (signal + behavior).

[0069] The alarm response issues alarms in levels. For example, yellow warning: only signal system abnormality (prompted by the dispatching center). Red strong warning: composite violation (triggering emergency braking + dispatching intervention).

[0070] The evidence chain record saves the ATO data stream segments (10 seconds before and after), iris videos, and grip sensor data for post - event analysis.

[0071] In some embodiments, refer to Figure 4 , Figure 4 is the flowchart of steps S401 - S403 provided by the embodiments of the present application, which will be described in combination with each step.

[0072] In step S401, lightweight blockchain nodes are deployed on the train on - vehicle terminal, and the driver operation logs, video key - frame hash values, and signal system status codes are encrypted and uploaded to the blockchain; In step S402, based on the encrypted data chain, the driver's biometric data is desensitized in a differential privacy manner, and only the desensitized trajectory vectors are uploaded after performing skeleton key - point detection locally; In step S403, the blockchain log is associated with the desensitized trajectory vector. When a timeout event of both hands leaving the console is detected, the track vibration spectrum data and the carriage monitoring video segment for the corresponding period are automatically retrieved to generate a verifiable multi-modal evidence chain.

[0073] Here, lightweight blockchain nodes are deployed on the train on-vehicle terminal by installing optimized lightweight blockchain client software. A lightweight consensus algorithm (such as a simplified version of the Practical Byzantine Fault Tolerance algorithm PBFT) is adopted to ensure the stable operation of the blockchain nodes without affecting the normal operation of the train. The network parameters of the blockchain nodes are configured so that they can communicate securely with other blockchain nodes in the train operation management system. By establishing an encrypted communication channel, the confidentiality and integrity of data transmission are guaranteed. During the train operation, the on-vehicle terminal collects the operation log data of the driver in real time, including information such as the operation time, operation type (such as acceleration, deceleration, gear shifting, etc.), and operation parameters. The symmetric encryption algorithm (such as the AES algorithm) is used to encrypt the operation log data, and the encryption key is generated and properly stored by the security module of the on-vehicle terminal. The encrypted operation log data is used as transaction data and submitted to the blockchain network through the interface of the blockchain node and recorded in the blocks of the blockchain.

[0074] The on-vehicle camera collects the video data inside the train carriage in real time and extracts the video key frames at a preset time interval. A hash calculation is performed on each key frame to generate a unique hash value. The asymmetric encryption algorithm (such as the RSA algorithm) is used to encrypt the hash value, and the encrypted hash value is also stored on the chain as transaction data. The train signal system sends a status code to the on-vehicle terminal in real time to indicate the running status of the train (such as normal running, signal failure, etc.). The on-vehicle terminal encrypts the received status code (the same symmetric encryption algorithm as the driver operation log can be used), and then uploads the encrypted status code to the chain.

[0075] Differential privacy desensitization processing deploys a differential privacy module locally on the on-vehicle terminal. When the biometric data of the driver (such as facial images, gesture actions, etc.) is collected, the differential privacy module first preprocesses the biometric data to extract the key features that can characterize the driver's identity and behavior.

[0076] Skeletal key point detection and desensitized trajectory vector upload utilize the pre-installed skeletal key point detection algorithm (such as a simplified version of the OpenPose algorithm) on the vehicle-mounted terminal to detect the driver's gesture actions in real time and obtain the skeletal key point coordinates of the driver's hands, arms and other parts. According to the desensitized skeletal key point coordinates, calculate the driver's gesture movement trajectory and represent it as a desensitized trajectory vector. This desensitized trajectory vector only contains the gesture movement trend information of the driver and does not contain sensitive information that can directly identify the driver's identity. The vehicle-mounted terminal uploads the desensitized trajectory vector to the server of the train operation management system through a secure communication channel for subsequent analysis and processing.

[0077] Data association On the server of the train operation management system, establish a blockchain log database and a desensitized trajectory vector database. Using the timestamp as the associated field, associate the data such as the driver's operation log, video key frame hash value, and signal system status code stored on the blockchain with the desensitized trajectory vector. For example, when an abnormal desensitized trajectory vector (such as both hands leaving the console) is detected at a certain time point, the driver's operation log, video key frame hash value, and signal system status code data near that time point can be found in the blockchain log database. Abnormal event detection and evidence chain generation Set a threshold for the timeout of both hands leaving the console (such as 5 seconds). When the system detects that the driver's both hands leave the console for more than this threshold, automatically trigger the abnormal event handling process. The system automatically retrieves the corresponding track vibration spectrum data (which can be obtained from the train track monitoring system) and the carriage monitoring video segment (retrieved from the video storage system according to the video key frame hash value) based on the associated data.

[0078] The generated multi-modal evidence chain is stored in a secure database and relevant managers are notified for processing. Managers can view the evidence chain to comprehensively understand the train operation status, driver operation situation, and surrounding environment information when the abnormal event occurs, providing strong support for subsequent investigations and decisions.

[0079] In some embodiments, refer to Figure 5 , Figure 5 which is a schematic flow diagram of steps S501 - S503 provided by the embodiments of this application and will be described in combination with each step.

[0080] In step S501, deploy a holographic projection module in front of the driver's console and construct a three-dimensional space mapping model of the operation interface through a ToF camera; In step S502, based on the three-dimensional space mapping model, when a non-standard gesture is recognized, generate a dynamic AR guiding light spot, and the color of the light spot gradually changes according to the error type, showing a yellow pulse when there is an operation delay and a red spiral ripple when the trajectory deviates; In step S503, when continuous operation errors are detected, a vibration prompt signal synchronized with the standard operation frequency is generated in the corresponding function key area.

[0081] Here, an edge blockchain node (such as a lightweight version of Hyperledger Fabric) is deployed on the train on-vehicle terminal, and only the data related to this train is stored, reducing the storage and computing overhead. The data uploaded to the blockchain includes the driver operation logs, which include console key operations, handle actions, etc., and are uploaded to the blockchain after being encrypted with SHA-256 hash. One frame is extracted every 10 seconds as a key frame (such as by the frame difference method of OpenCV), and its hash value is calculated and uploaded to the blockchain. The signal system status codes, such as the ATO / ATP system status (such as mode switching, braking instructions), are encrypted and uploaded to the blockchain according to the time stamp.

[0082] A simplified version of Practical Byzantine Fault Tolerance (PBFT) is adopted, allowing only the on-vehicle terminal and the nodes of the adjacent two carriages to participate in the consensus, reducing the communication delay (the target consensus time < 200ms), and using the Merkle tree structure to store the log hash, reducing the storage space (only the root hash needs to be stored for every 1000 logs).

[0083] For the desensitization processing of biometric data, in the on-vehicle terminal of this application embodiment, MediaPipe or OpenPose is used to extract 25 key points of the driver's upper body (such as shoulders, elbows, wrists, etc.), generating an original trajectory vector (dimension = 25×3 = 75, including x / y coordinates and confidence). Laplace noise (LaplaceMechanism) is added to each coordinate in the trajectory vector, and the privacy budget ε = 0.5 (meeting ε-differential privacy).

[0084] The multi-modal evidence chain is specifically generated in the following way: When the hands leave the console for an overtime, based on the desensitized trajectory vector, the speeds (v = Δd / Δt) and accelerations (a = Δv / Δt) of the key points of the two wrists are calculated. The trigger condition is that the hands' speeds are continuously < 0.1m / s (stationary) for 3 seconds and the acceleration is close to 0 (no tiny movements).

[0085] The evidence chain association is based on the blockchain log query. According to the abnormal event time stamp, the following are retrieved from the blockchain for the previous and next 10 seconds: the driver operation log hash value, the video key frame hash value (used to verify the integrity of the monitoring video), and the signal system status code (such as whether it is in the manual driving mode).

[0086] Rail vibration and video retrieval include rail vibration spectrum data: Obtain Z-axis vibration data (sampling rate 1 kHz) from on-vehicle vibration sensors (such as accelerometer ADXL355), calculate the spectrum (0 - 100 Hz) through FFT, and carriage monitoring video segments: Extract 30 seconds of monitoring video (resolution 720p) before and after the abnormal period to verify whether the driver is talking to others or distracted.

[0087] For the verification of the evidence chain, in the embodiments of the present application, by comparing the hash values of the key frames of the video stored in the blockchain with the hash values of the actual video segments, it is ensured that the data has not been tampered with. A multi-modal Transformer model is used to fuse the vibration spectrum, video segments, and desensitized trajectory vectors to generate an abnormal behavior confidence level (0 - 1).

[0088] In some embodiments, refer to Figure 6 , Figure 6 is the flow schematic diagram of steps S601 - S603 provided by the embodiments of the present application, which will be described in combination with each step.

[0089] In step S601, a subway environmental parameter matrix is established, and the subway environmental parameter matrix includes rail friction coefficient and pantograph - catenary current data; In step S602, an environmental perception adversarial network EAA - Net is constructed based on the environmental parameter matrix. When detecting that the rail is wet and slippery, the sampling frequency of the skeleton key point tracking algorithm is increased; In step S603, the rotation invariance parameter of the gesture recognition model is dynamically adjusted, and the weight decay coefficient of the fully connected layer of the action classification model is increased based on the rail friction coefficient.

[0090] Here, parameter definition and acquisition are first performed. The data includes: (1) Rail friction coefficient.

[0091] Sensor: On - vehicle friction coefficient measuring instrument (such as Wheel - Rail Force Transducer), calculate the μ value through the wheel - rail contact force and normal pressure.

[0092] Data classification: The μ value is divided into three levels: dry (μ > 0.4), wet and slippery (0.2 < μ ≤ 0.4), and frozen (μ ≤ 0.2).

[0093] (2) Pantograph - catenary current: Sensor: Hall current sensor (such as LAH 50 - P), measurement range 0 - 500 A, sampling frequency 1 kHz.

[0094] Feature extraction: Calculate the current volatility (σ / μ, where σ is the standard deviation and μ is the mean) to evaluate the stability of the catenary.

[0095] Align the rail friction coefficient and catenary current according to the time stamp, construct a three-dimensional environmental parameter matrix, and perform anomaly detection on the environmental parameter matrix based on LSTM-Autoencoder. Set the threshold to trigger an alarm when the reconstruction error > 0.15.

[0096] The Environment Awareness Adversarial Network (EAA-Net) is based on an improved structure of U-Net, and the depth of the convolutional kernel can be dynamically adjusted. The Skeleton Key Point Tracking Module is based on a lightweight version of AlphaPose and supports dynamic adjustment of the sampling frequency. The Gesture Recognition and Action Classification Module is based on an improved network of 3D-ResNet and supports dynamic adjustment of the rotation invariance parameter and the weight decay coefficient.

[0097] When the rail friction coefficient μ ≤ 0.4, increase the sampling frequency of the skeleton key point tracking algorithm from 15 Hz to 30 Hz through a Dynamic Frame Rate Controller.

[0098] Adjust the rotation invariance parameter for the curve curvature and gesture recognition. Calculate the curve curvature κ (unit: 1 / m) through an in-vehicle Inertial Measurement Unit (IMU). When κ > 0.01 (sharp curve), expand the rotation invariance parameter θ of the gesture recognition model from ±15° to ±30° by inserting a Learnable Rotation Layer after the convolutional layer of 3D-ResNet; when the rail friction coefficient μ ≤ 0.2 (icing), the weight decay coefficient λ of the fully connected layer of the action classification model can be increased from 0.001 to 0.01. It can be achieved by dynamically modifying the L2 regularization term in the optimizer (such as Adam).

[0099] In some embodiments, refer to Figure 7 , Figure 7 is a schematic flow diagram of steps S701 - S703 provided by the embodiments of the present application, which will be described in conjunction with each step.

[0100] In step S701, integrate a quantum key distribution module in the in-vehicle terminal to generate a true random encryption seed based on the track position; In step S702, based on the quantum key, use a post-quantum cryptography algorithm to perform block encryption on the driver operation video stream, and perform frame extraction analysis on the key action frames in the ciphertext domain; In step S703, based on the encryption seed and tunnel electromagnetic interference data, through a noise adaptive injection mechanism, superimpose matching noise during wireless transmission.

[0101] Here, a continuous variable quantum key distribution (CV-QKD) scheme is adopted. Based on the Gaussian modulated coherent state (GMCS) protocol, a key is generated between the on-vehicle terminal and the quantum base station beside the track. The real-time data of the track vibration sensor (such as the accelerometer ADXL355) is used as a physical entropy source, and combined with the randomness extraction algorithm (such as Toeplitz hashing) of the QKD protocol, a truly random encryption seed is generated. At a train speed of 80 km / h, the target key generation rate is ≥100 kbps, meeting the real-time encryption requirements of video streams.

[0102] A new key is generated every 10 seconds, and the key version number and generation timestamp are recorded through a blockchain light node (deployed on the on-vehicle terminal) to ensure the traceability of the key. A key encapsulation mechanism based on lattice-based cryptography (such as the Kyber algorithm) is adopted to resist quantum computing attacks by the Shor algorithm.

[0103] In summary, the embodiments of the present application have the following beneficial effects: (1) Through the fusion of multi-modal data (video, sensors, positioning, etc.) and dynamic analysis algorithms, accurate identification of abnormal behaviors such as the driver leaving the seat, gesture operations, and distracted attention is achieved, effectively reducing the false alarm rate.

[0104] (2) Based on edge computing and quantum-secure encryption technology, it ensures anti-interference in real-time data processing and transmission, and still maintains a high system availability in complex electromagnetic environments (such as tunnels).

[0105] (3) Automatically trigger hierarchical alarms (such as yellow warnings and red strong warnings), associate video clips with sensor data, with a small response delay, and support rapid intervention by dispatchers.

[0106] (4) Adopt quantum key distribution and post-quantum cryptographic algorithms to ensure the security of data transmission; combine blockchain and differential privacy technologies to achieve the non-tampering of operation logs and biometric desensitization.

[0107] (5) Through the Environment-Aware Adversarial Network (EAA-Net), dynamically adjust the image enhancement and gesture recognition parameters to adapt to complex scenarios such as tunnel dust and slippery rails, effectively improving the recognition accuracy.

[0108] (6) Cover the entire process of vehicle entering the station, driver inspection, and leaving the station operation, and combine AR guidance and vibration prompts to reduce the risk of human error and effectively improve the accident prevention efficiency.

[0109] Based on the same inventive concept, an apparatus for identifying abnormal behaviors of subway drivers based on multimodal data fusion corresponding to the method for identifying abnormal behaviors of subway drivers based on multimodal data fusion in the first embodiment is further provided in the embodiments of the present application. Since the principle of solving problems by the apparatus in the embodiments of the present application is similar to the above-mentioned method for identifying abnormal behaviors of subway drivers based on multimodal data fusion, the implementation of the apparatus can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0110] As Figure 8 shown, Figure 8 FIG. 7 is a schematic structural diagram of an apparatus 800 for identifying abnormal behaviors of subway drivers based on multimodal data fusion provided by an embodiment of the present application. The apparatus 800 for identifying abnormal behaviors of subway drivers based on multimodal data fusion includes: An inbound confirmation module 801, configured to analyze the door opening / closing sensor and the platform screen door status signal, and generate a vehicle inbound confirmation instruction when the doors and the platform screen doors are opened synchronously and the static background flag is valid; wherein, the static background flag is triggered when the detected background motion vector is lower than a threshold and lasts for a first specific time; A patrol determination module 802, configured to, when the seat pressure distribution sensor detects that the pressure value is lower than a set threshold for more than a second specific time, extract the driver's skeletal key points through an infrared thermal imaging camera and a key point extraction model, and calculate the offset of the torso centroid trajectory. If the centroid lateral offset exceeds a specific distance and lasts for a third specific time, it is determined that the driver leaves the seat to perform a patrol operation; An outbound confirmation module 803, configured to continuously monitor the track circuit signal status, and when receiving an ATO outbound instruction from the Automatic Train Supervision system, start dynamic background analysis and synchronously verify the door and platform screen door closing signals. When the motion vector continuously increases and the door status is closed, generate a vehicle outbound completion event; An abnormal alarm module 804, configured to analyze the console touch pressure matrix data. When it is detected that the interval between N consecutive pressure peaks is less than a fourth specific time, it is determined that the standard hand gesture on the console is completed. Based on a multispectral camera to capture the hand contour, the contact area and the spatial angle between the hand and the console are calculated in real time through a hand key point three-dimensional reconstruction model. If the contact area is less than a specific proportion or the angle deviates from a specific degree for more than a fourth specific time, an abnormal alarm is triggered; An aggregation and transmission module 805, configured to aggregate inbound / outbound events, patrol status, and action recognition results at an edge computing node, generate a structured log, and encrypt and transmit the data to a central dispatching system through a 5G dedicated network, and dynamically map the driver behavior status in a digital twin cockpit. When an abnormality is detected in any link, a hierarchical alarm is automatically triggered, and the hierarchical alarm at least includes a yellow warning for operation delay display, a red strong warning for safety violation display, and association of corresponding video clips and sensor data snapshots.

[0111] Those skilled in the art should understand that Figure 8 The implementation functions of the units in the subway driver abnormal behavior recognition device 800 based on multimodal data fusion shown can be understood with reference to the relevant descriptions of the aforementioned subway driver abnormal behavior recognition method based on multimodal data fusion. Figure 8 The functions of the units in the subway driver abnormal behavior recognition device 800 based on multimodal data fusion shown can be realized by a program running on a processor or by specific logic circuits.

[0112] In a possible implementation manner, the method further includes: Construct a subway line feature database to record parameters such as the length of each platform, the curvature of the bend, and the tunnel illumination intensity; Based on the line feature database, design a multi-task learning network, where the main branch of the multi-task learning network is used to analyze the spatio-temporal continuity of the driver's gesture trajectory, and the auxiliary branch is used to dynamically adjust the image enhancement parameters according to the real-time tunnel illumination data; Based on the bend curvature, braking distance parameter, the current speed of the train, and the remaining length of the platform, adaptively correct the gesture recognition decision time window, and automatically extend the action verification duration to a specific multiple of the standard value when the train enters the tunnel; where the specific multiple is greater than 1 times and less than or equal to 1.3 times.

[0113] In a possible implementation manner, the method further includes: Real-time parse the data stream of the ATO to obtain the train operation mode, emergency braking state, and track occupancy information; Based on the data stream of the ATO, when the train is in the manual driving mode, activate the multispectral camera to collect the driver's iris features, and synchronously extract the data of the grip force sensor on the console to construct an attention concentration scoring model, where the attention concentration scoring model is used to output an attention score; Based on the track occupancy information and the attention score, when it is detected that the ATO ready signal is not received after the train leaves the station and the driver's attention score is lower than the threshold, trigger a composite alarm event that combines signal system anomalies and behavior violations.

[0114] In a possible implementation manner, the method further includes: Deploy lightweight blockchain nodes on the train on-board terminal, and encrypt and upload the driver operation logs, video key frame hash values, and signal system status codes to the blockchain; Based on the encrypted data chain, desensitize the driver's biometric data in a differential privacy manner, and only upload the desensitized trajectory vector after performing skeleton key point detection locally; Associate the blockchain log with the desensitized trajectory vector. When a timeout event of both hands leaving the console is detected, automatically retrieve the track vibration spectrum data and the carriage monitoring video clip for the corresponding period, and generate a verifiable multi-modal evidence chain.

[0115] In a possible implementation manner, the method further includes: Deploy a holographic projection module in front of the driver's console, and construct a three-dimensional space mapping model of the operation interface through a ToF camera; [[ID=~]]Based on the three-dimensional space mapping model, when a non-standard gesture is recognized, generate a dynamic AR guiding light spot, and the color of the light spot changes gradually according to the error type, showing a yellow pulse when there is an operation delay, and a red spiral ripple when the trajectory deviates; When continuous operation errors are detected, generate a vibration prompt signal synchronized with the standard operation frequency in the corresponding function key area.

[0116] In a possible implementation manner, the method further includes: Establish a subway environmental parameter matrix, including tunnel dust concentration, rail friction coefficient, and pantograph-catenary current data; Based on the environmental parameter matrix, construct an environmental perception adversarial network EAA-Net, automatically enhance the convolution kernel depth of the image defogging module according to the real-time dust concentration, and increase the sampling frequency of the skeleton key point tracking algorithm when rail slipperiness is detected; Based on the curve curvature data, dynamically adjust the rotation invariance parameter of the gesture recognition model, and increase the full connection layer weight decay coefficient of the action classification model based on the rail friction coefficient.

[0117] In a possible implementation manner, the method further includes: Integrate a quantum key distribution module in the vehicle-mounted terminal to generate a true random encryption seed based on the track position; Based on the quantum key, use a post-quantum cryptography algorithm to block-encrypt the driver operation video stream, and perform frame extraction analysis on the key action frames in the ciphertext domain; Based on the encryption seed and the tunnel electromagnetic interference data, through a noise adaptive injection mechanism, superimpose matching noise during wireless transmission.

[0118] The above-mentioned subway driver abnormal behavior recognition device based on multi-modal data fusion has the following beneficial effects: (1) Through multi-modal data (video, sensors, positioning, etc.) fusion and dynamic analysis algorithms, accurate recognition of abnormal behaviors such as the driver leaving the seat, gesture operations, and distracted attention is achieved, effectively reducing the false alarm rate.

[0119] (2)Based on edge computing and quantum-secure encryption technology, ensure real-time data processing and anti-interference transmission, and maintain high system availability in complex electromagnetic environments (such as tunnels).

[0120] (3)Automatically trigger hierarchical alarms (such as yellow warnings and red strong warnings), correlate video clips with sensor data, have a small response delay, and support rapid intervention by dispatchers.

[0121] (4)Adopt quantum key distribution and post-quantum cryptographic algorithms to ensure data transmission security; combine blockchain and differential privacy technologies to achieve the immutability of operation logs and biometric desensitization.

[0122] (5)Through the Environment Awareness Adversarial Network (EAA-Net), dynamically adjust image enhancement and gesture recognition parameters to adapt to complex scenarios such as tunnel dust and slippery rails, and effectively improve the recognition accuracy.

[0123] (6)Cover the entire process such as vehicle entry, driver inspection, and exit operations, and combine AR guidance and vibration prompts to reduce the risk of human error and effectively improve the accident prevention efficiency.

[0124] Such as Figure 9 shown, Figure 9 is a schematic diagram of the composition structure of the electronic device 900 provided by the embodiment of the present application. The electronic device 900 includes: A processor 901, a storage medium 902, and a bus 903. The storage medium 902 stores machine-readable instructions executable by the processor 901. When the electronic device 900 runs, the processor 901 communicates with the storage medium 902 through the bus 903. The processor 901 executes the machine-readable instructions to perform the steps of the method for recognizing abnormal behaviors of subway drivers based on multi-modal data fusion according to the embodiment of the present application.

[0125] In practical applications, the various components in the electronic device 900 are coupled together through the bus 903. It can be understood that the bus 903 is used to realize the connection and communication between these components. In addition to the data bus, the bus 903 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 9 all kinds of buses are labeled as the bus 903.

[0126] The above-mentioned electronic device has the following beneficial effects: (1)Through the fusion of multi-modal data (video, sensors, positioning, etc.) and dynamic analysis algorithms, accurately identify abnormal behaviors such as the driver leaving the seat, gesture operations, and distraction of attention, effectively reducing the false alarm rate.

[0127] (2) Based on edge computing and quantum security encryption technology, ensure real-time data processing and anti-interference transmission, and maintain high system availability in complex electromagnetic environments (such as tunnels).

[0128] (3) Automatically trigger hierarchical alarms (such as yellow warnings and red strong warnings), associate video clips with sensor data, with a small response delay, and support rapid intervention by dispatchers.

[0129] (4) Adopt quantum key distribution and post-quantum cryptographic algorithms to ensure data transmission security; combine blockchain and differential privacy technologies to achieve the immutability of operation logs and biometric desensitization.

[0130] (5) Through the Environment Aware Adversarial Network (EAA-Net), dynamically adjust image enhancement and gesture recognition parameters to adapt to complex scenarios such as tunnel dust and slippery rails, and effectively improve the recognition accuracy.

[0131] (6) Cover the entire process of vehicle entry, driver inspection, and exit operations, and combine AR guidance and vibration prompts to reduce the risk of human error and effectively improve the accident prevention efficiency.

[0132] The embodiment of the present application also provides a computer-readable storage medium, and the storage medium stores executable instructions. When the executable instructions are executed by at least one processor 901, the method for identifying abnormal behaviors of subway drivers based on multi-modal data fusion described in the embodiment of the present application is implemented.

[0133] In some embodiments, the storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it may also be various devices including one or any combination of the above memories.

[0134] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as a stand-alone program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0135] As an example, the executable instructions may or may not correspond to a file in the file system, and may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, stored in a single file dedicated to the program in question, or, stored in multiple cooperating files (such as files that store one or more modules, subroutines, or portions of code).

[0136] As an example, the executable instructions may be deployed to execute on one computing device, or on multiple computing devices located at one site, or, on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0137] The above computer-readable storage medium has the following beneficial effects: (1) Through multimodal data (video, sensors, positioning, etc.) fusion and dynamic analysis algorithms, accurate identification of abnormal behaviors such as the driver leaving the seat, gesture operations, and distracted attention is achieved, effectively reducing the false alarm rate.

[0138] (2) Based on edge computing and quantum-secure encryption technology, it ensures real-time data processing and transmission anti-interference, and still maintains high system availability in complex electromagnetic environments (such as tunnels).

[0139] (3) Automatically triggers hierarchical alarms (such as yellow warnings, red strong warnings), associates video clips with sensor data, has a small response delay, and supports rapid intervention by dispatchers.

[0140] (4) Adopts quantum key distribution and post-quantum cryptographic algorithms to ensure data transmission security; combines blockchain and differential privacy technologies to achieve the non-tampering of operation logs and biometric desensitization.

[0141] (5) Through the Environment Awareness Adversarial Network (EAA-Net), dynamically adjusts image enhancement and gesture recognition parameters to adapt to complex scenarios such as tunnel dust and slippery rails, effectively improving the recognition accuracy.

[0142] (6) Covers the entire process of vehicle entering the station, driver inspection, and station exit operations, combines AR guidance and vibration prompts, reduces the risk of human error, and effectively improves the accident prevention efficiency.

[0143] In several embodiments provided in the present application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces. The indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0144] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0146] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0147] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for identifying abnormal behaviors of subway drivers based on multi-modal data fusion, characterized in that, The method includes: Analyze the door opening / closing sensor and the platform screen door status signal, and generate a vehicle approaching station confirmation instruction when the doors and the platform screen doors are opened synchronously and the static background flag is valid; wherein, the static background flag is triggered when the detected background motion vector is lower than the threshold and lasts for the first specific time; When the seat pressure distribution sensor detects that the pressure value is lower than the set threshold for more than the second specific time, extract the driver's skeletal key points through an infrared thermal imaging camera and a key point extraction model, and calculate the offset of the torso centroid trajectory. If the centroid laterally deviates by more than a specific distance and lasts for the third specific time, it is determined that the driver leaves the seat to perform a patrol operation; Monitor the track circuit signal status in real time. When receiving the ATO (Automatic Train Operation) outbound instruction, start dynamic background analysis and synchronously verify the door and platform screen door closed signals. When the motion vector continuously increases and the door status is closed, generate a vehicle outbound completion event; Analyze the console touch pressure matrix data. When it is detected that the interval between N consecutive pressure peaks is less than the fourth specific time, it is determined that the standard gesture on the console is completed. Based on the multi-spectral camera to capture the hand contour, the contact area and the spatial angle between the hand and the console are calculated in real time through the hand key point three-dimensional reconstruction model. If the contact area is less than a specific proportion or the angle deviates from a specific degree for more than the fourth specific time, an abnormal alarm is triggered; Aggregate the inbound / outbound events, patrol status, and action recognition results at the edge computing node to generate a structured log, and encrypt and transmit the data to the central dispatching system through a 5G private network, and dynamically map the driver's behavior status in the digital twin cockpit. When any abnormality is detected in any link, a hierarchical alarm is automatically triggered. The hierarchical alarm at least includes a yellow warning for operation delay display, a red strong warning for safety violation display, and the association of the corresponding video clip and the sensor data snapshot.

2. The method according to claim 1, wherein The method further includes: Construct a subway line feature database to record the length of each platform, the curvature of the curve, and the tunnel light intensity parameters; Based on the line feature database, design a multi-task learning network. Among them, the main branch of the multi-task learning network is used to analyze the spatio-temporal continuity of the driver's gesture trajectory, and the auxiliary branch is used to dynamically adjust the image enhancement parameters according to the real-time tunnel light data; Based on the curve curvature, braking distance parameter, the current speed of the train, and the remaining length of the platform, adaptively correct the gesture recognition decision time window, and automatically extend the action verification duration to a specific multiple of the standard value when the train enters the tunnel; wherein, the specific multiple is greater than 1 times and less than or equal to 1.3 times.

3. The method according to claim 1, characterized in that The method further includes: Parse the data stream of the ATO in real time to obtain the train operation mode, emergency braking status, and track occupancy information; Based on the data stream of the ATO, when the train is in the manual driving mode, activate the multi-spectral camera to collect the driver's iris features, and synchronously extract the console grip sensor data to construct an attention concentration scoring model, wherein the attention concentration scoring model is used to output an attention score; Based on the track occupancy information and the attention score, when it is detected that the ATO ready signal has not been received after the train departs the station and the driver's attention score is lower than the threshold, a composite alarm event that combines signal system anomalies and behavior violations is triggered.

4. The method according to claim 1, wherein The method further includes: Deploy lightweight blockchain nodes on the train on-board terminal, and encrypt and upload the driver operation log, video key frame hash value, and signal system status code to the blockchain; Based on the encrypted data chain, desensitize the driver's biometric data in a differential privacy manner, and only upload the desensitized trajectory vector after performing skeleton key point detection locally; Associate the blockchain log with the desensitized trajectory vector. When a timeout event of both hands leaving the console is detected, automatically retrieve the track vibration spectrum data and carriage monitoring video segments for the corresponding period, and generate a verifiable multi-modal evidence chain.

5. The method according to claim 1, wherein The method further includes: Deploy a holographic projection module in front of the driver console, and construct a three-dimensional space mapping model of the operation interface through a ToF camera; Based on the three-dimensional space mapping model, when a non-standard gesture is recognized, generate a dynamic AR guiding light spot, and the color of the light spot fades according to the error type, showing a yellow pulse when there is an operation delay, and a red spiral ripple when the trajectory deviates; When a continuous operation error is detected, generate a vibration prompt signal synchronized with the standard operation frequency in the corresponding function key area.

6. The method according to claim 1, wherein The method further includes: Establish a subway environmental parameter matrix, where the subway environmental parameter matrix includes the rail friction coefficient and the pantograph catenary current data; Based on the environmental parameter matrix, construct an environmental perception adversarial network EAA-Net, and increase the sampling frequency of the skeleton key point tracking algorithm when it is detected that the rail is slippery; Based on the curve curvature data, dynamically adjust the rotation invariance parameter of the gesture recognition model, and increase the full connection layer weight decay coefficient of the action classification model based on the rail friction coefficient.

7. The method according to claim 1, characterized in that The method further includes: Integrate a quantum key distribution module on the on-board terminal to generate a truly random encryption seed based on the track position; Based on the quantum key, use a post-quantum cryptography algorithm to block-encrypt the driver operation video stream, and perform frame extraction analysis on the key action frames in the ciphertext domain; Based on the encryption seed and the tunnel electromagnetic interference data, through a noise adaptive injection mechanism, superimpose matching noise during wireless transmission.

8. An abnormal behavior recognition device for subway drivers based on multi-modal data fusion, characterized in that, The device includes: An inbound confirmation module, configured to analyze the door opening and closing sensor and the platform screen door status signal, and generate a vehicle inbound confirmation instruction when the doors and the platform screen doors are opened synchronously and the static background flag is valid; wherein, the static background flag is triggered when the detected background motion vector is lower than the threshold and lasts for a first specific time; A patrol determination module, configured to, when the seat pressure distribution sensor detects that the pressure value is below the set threshold for more than a second specific time, extract the driver's skeleton key points through an infrared thermal imaging camera and a key point extraction model, and calculate the offset of the torso centroid trajectory. If the centroid lateral offset exceeds a specific distance and lasts for a third specific time, determine that the driver leaves the seat to perform a patrol operation; An outbound confirmation module is used to monitor the status of track circuit signals in real time. When receiving the ATO outbound instruction from the Automatic Train Monitoring System, it starts dynamic background analysis and synchronously verifies the door and platform screen door closed signals. When the motion vector continues to increase and the door status is closed, it generates a vehicle outbound completion event; An abnormal alarm module is used to analyze the console touch pressure matrix data. When it detects that the interval between N consecutive pressure peaks is less than the fourth specific time, it determines that the standard gesture on the console is completed. Based on the multi-spectral camera to capture the hand contour, it calculates the contact area and spatial angle between the hand and the console in real time through the three-dimensional reconstruction model of hand key points. If the contact area is less than a specific proportion or the angle deviates from a specific degree for more than the fourth specific time, it triggers an abnormal alarm; An aggregation transmission module is used to aggregate inbound / outbound events, patrol status, and action recognition results at the edge computing node, generate structured logs, and encrypt and transmit the data to the central dispatching system through a 5G private network, and dynamically map the driver's behavior status in the digital twin cockpit. When any abnormality is detected in any link, it automatically triggers hierarchical alarms, and the hierarchical alarms at least include a yellow warning for operation delay display, a red strong warning for safety violations and associate the corresponding video clips and sensor data snapshots.

9. An electronic device, characterized in that, It includes: A processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus. The processor executes the machine-readable instructions to execute the method for identifying abnormal behaviors of subway drivers based on multi-modal data fusion according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is run by the processor, it executes the method for identifying abnormal behaviors of subway drivers based on multi-modal data fusion according to any one of claims 1 to 7.

Images