A railway driver driving behavior monitoring method based on artificial intelligence

Abstract

A kind of railway driver driving behavior monitoring method based on artificial intelligence, artificial intelligence processing unit completes hardware interface initialization;Railway locomotive comprehensive wireless device sends the network IP address of main control end, device identification information to the main process of artificial intelligence processing unit;Artificial intelligence processing unit main process initiates video stream pull request to the digital camera of corresponding end by RTSP protocol;Artificial intelligence processing unit main process decodes into JPEG format picture after receiving RTSP code stream, trims zoom image, adds complete metadata information to each picture;Three inference processes of artificial intelligence processing unit run in parallel, each process is based on the model loaded by itself to decode the JPEG picture and metadata for targeted inference detection:Main process receives inference detection result, carries out comprehensive judgment in conjunction with preset threshold and abnormal state duration, realizes local reminder, driver notification, remote monitoring triple alarm, standardizes driver driving behavior.

Description

Technical Field

[0001] This invention relates to an artificial intelligence-based method for monitoring the driving behavior of railway drivers. It belongs to the interdisciplinary technical field of artificial intelligence image recognition, video processing and railway traffic safety monitoring. It is applicable to the real-time monitoring of the driving status of drivers of various locomotives on conventional and high-speed railways. It can accurately identify and promptly warn drivers of violations such as leaving their seats, using mobile phones, smoking, and driving while fatigued. Background Technology

[0002] With the rapid expansion of my country's railway transportation network and the continuous increase in train operating speed, railway traffic safety places higher demands on driver condition monitoring. Train drivers need to maintain a high level of concentration for extended periods to cope with complex and ever-changing operating environments. However, the monotonous and enclosed driving environment can easily induce fatigue, inattention, and "microsleep" (brief lapses in concentration), leading to serious traffic accidents.

[0003] The current driver status monitoring methods commonly used in the railway sector have the following problems and deficiencies: 1. Traditional physical alert devices: These devices, based on the standard TB / T 3333-2022 "Technical Conditions for Driver Cab Equipment Alertness", require drivers to periodically (usually 30-60 seconds) press a pedal or button to confirm their alertness. Their core flaw is that they only verify whether the driver performs the action, not their true physiological state. For example, drivers may mechanically complete button operations even with closed eyes, drowsy, or distracted, making it difficult for the device to reveal the actual level of fatigue. Furthermore, this method is easily circumvented, failing to fundamentally improve driving safety.

[0004] 2. Early Detection Devices: Early visual fatigue monitoring systems, while incorporating computer vision technology, typically rely on a single monitoring dimension, primarily focusing on eye features such as eye closure or blink frequency. However, the accuracy and precision of current technologies for judging driver fatigue are often low. Some existing solutions, such as CN120918657A (a portable fatigue driving monitoring and early warning system), use EEG recognition, which lacks universality; CN120932210A (a driver fatigue state early warning method and system based on face recognition tracking), use semantic segmentation to extract face and eye features, but the detection method is limited, requires significant computing power, and is difficult to deploy on edge devices.

[0005] 3. Insufficient monitoring capabilities for driver misconduct: Railway drivers face increasing work pressure, and prolonged driving can easily lead to fatigue, inattention, or lack of focus, even resulting in non-compliant behaviors such as making phone calls, leaving their seats, driving while fatigued, and smoking. These behaviors not only threaten railway operational safety but may also cause serious traffic accidents. Currently, although some monitoring systems can detect drivers' working status in real time, most are based on simple environmental monitoring and lack in-depth identification and analysis of driver behavior, or rely on manual review to determine whether the driver has violated driving regulations. Existing technologies are insufficient to effectively monitor and identify behaviors such as fatigue, leaving one's seat, using mobile phones, and smoking. Therefore, a multi-dimensional, real-time, and robust driver status monitoring solution is needed to reduce false alarms and missed alarms, improve system reliability, and effectively ensure railway traffic safety in complex railway driving environments. Summary of the Invention

[0006] In view of the requirements for railway driver status detection and safe and compliant driving, and to address the shortcomings of existing technologies in monitoring railway driver behavior, this invention proposes an artificial intelligence-based method for monitoring railway driver driving behavior. This method utilizes existing integrated radio equipment (CIR) for railway locomotives combined with a domestically produced artificial intelligence inference chip solution, and based on various image recognition algorithms and detection and analysis technologies, to achieve real-time monitoring and anomaly alarms of driver behavior, ensuring standardized operation, preventing safety hazards caused by non-compliant driver behavior, providing strong protection for railway transportation safety, and improving the safety of railway operations.

[0007] The technical solution adopted in this invention is: a railway driver driving behavior monitoring method based on artificial intelligence, which is implemented based on an artificial intelligence processing unit and a railway locomotive integrated wireless device. The steps are as follows: Step 1, after the artificial intelligence processing unit is powered on and running, it first completes the hardware interface initialization, operating system environment configuration, log and artificial intelligence algorithm model loading.

[0008] Step 2: After the driver completes the main control terminal selection operation through the MMI human-machine interaction terminal at the front or rear of the train, the railway locomotive integrated wireless equipment sends the network IP address and device identification information of the main control terminal to the main process of the artificial intelligence processing unit in real time.

[0009] Step 3: The main process of the artificial intelligence processing unit initiates a video stream retrieval request to the corresponding digital camera via the RTSP protocol based on the received network address of the master control terminal.

[0010] Step four: After receiving the request, the digital camera continuously pushes the RTSP stream to the main process of the artificial intelligence processing unit through the data channel. After receiving the RTSP stream, the main process of the artificial intelligence processing unit decodes the stream into JPEG format images in real time through the acceleration module, crops and scales the images, and uses them as input for inference of various models; and adds complete metadata information to each image.

[0011] Step 5: The three inference processes of the artificial intelligence processing unit run in parallel. Each process performs targeted inference and detection on the decoded JPEG image and metadata based on its own loaded model, and then transmits it to the main process.

[0012] Step six: The main process receives and integrates the inference detection results from each inference process. For different abnormal behavior types, it makes a comprehensive judgment based on preset thresholds and the duration of abnormal states, thereby achieving triple alarms: local reminders, driver notifications, and remote monitoring, to regulate driver behavior.

[0013] The specific method for step one is as follows: Initialize the hardware interface and the software runtime environment; start the main process of the artificial intelligence processing unit acceleration module and three inference processes: target detection process, fatigue detection process, and motion detection process; initialize the resource management and memory management environment for the artificial intelligence runtime in the inference process, and load the corresponding trained artificial intelligence models to support inference calculations for different categories of behaviors.

[0014] Step four is as follows: The main process receives the RTSP stream from the camera of the artificial intelligence processing unit and decodes it into a JPEG image through the acceleration module of the artificial intelligence processing unit; the JPEG image is preprocessed by cropping, scaling, normalizing, and frame extraction, and used as data input for the subsequent artificial intelligence model; metadata information is added to realize the association between the image and the train's status. The metadata information includes the millisecond-level timestamp of the local system, the standard time from the integrated wireless equipment of the railway locomotive, the precise latitude and longitude position, the speed of travel, and the identification of the main control device, to ensure the traceability of each frame of image. After the main process serializes the above three parts of data (decoding, preprocessing, and metadata addition), it publishes them to the inference process through the inter-process communication mechanism.

[0015] The specific method for step five is as follows: the target detection process, fatigue detection process, and motion detection process each perform the following artificial intelligence processing functions: the target detection process identifies people, faces, mobile phones, and cigarettes in the image; the fatigue detection process performs artificial intelligence processing for face detection, liveness detection, face marker detection, and eye-open / closed classification; the motion detection process analyzes the movement of people; and each detection process transmits the inference results, including target category, location, confidence level, key point coordinates, and state determination results, to the main process in real time through an inter-process communication mechanism.

[0016] The specific method for step six is ​​as follows: The main process receives and integrates the inference detection results of each inference process, and makes a comprehensive judgment based on the preset threshold and the duration of the abnormal state to form alarm types such as absence alarm, mobile phone use alarm, smoking alarm, fatigue alarm, and liveness detection alarm. If the alarm conditions are met, the main program immediately triggers three types of responses: First, it controls the sound and light alarm device to issue a sound and light prompt through the digital IO port; second, it sends the alarm information to the MMI human-machine interaction terminal for display via the UDP protocol; and third, it reports alarm images and information to the ground center in real time via the wireless communication unit of the railway locomotive integrated wireless equipment through the UDP protocol.

[0017] The technical effects of this invention are: 1. Strong real-time performance: Using Huawei Atlas 200I A2 hardware acceleration, the single-frame image processing time is ≤100ms and the alarm response delay is ≤0.5 seconds, which is far superior to the 1~3 seconds response time of the existing technology; the frame extraction processing mechanism balances performance and accuracy, meeting the requirements of 24-hour continuous operation.

[0018] 2. High accuracy: Through multimodal fusion algorithms and personalized threshold judgment, the impact of environmental interference and individual differences is effectively reduced. Actual test data shows that the accuracy of recognizing behaviors such as leaving the seat, using a mobile phone, and smoking is ≥95%, and the accuracy of recognizing fatigue driving is ≥95%, which is a significant improvement over existing technologies (false alarm rate of more than 10% and false alarm rate of more than 5%).

[0019] 3. Good compatibility: Based on the integration of existing railway locomotive integrated wireless equipment, there is no need to add new communication modules or make major modifications to the locomotive, reducing construction costs and modification risks. It can be adapted to more than 90% of existing railway locomotives (including Harmony and Fuxing series EMUs and conventional locomotives).

[0020] 4. Comprehensive functions: It covers five major scenarios: leaving the seat, using a mobile phone, smoking, fatigue driving, and anti-facial recognition, solving the problem of the single monitoring dimension of existing technologies; it supports personalized threshold configuration (such as adjusting the fatigue judgment time for different routes) to adapt to different operational needs. Attached Figure Description

[0021] Figure 1 The overall system structure diagram for implementing the present invention;

[0022] Figure 2 The flowchart of the behavior monitoring method of the present invention is shown below;

[0023] Figure 3 This is a schematic diagram of the face marker points of the present invention;

[0024] Figure 4 This is a schematic diagram of the detection frame in the target detection of the present invention. Detailed Implementation

[0025] To more clearly illustrate the technical solutions and implementation examples of this invention, the invention will be further described in detail below with reference to the accompanying drawings.

[0026] Figure 1 To realize the system connection diagram of the present invention, it includes an artificial intelligence processing unit (DMS) and a railway locomotive integrated wireless device (CIR). The artificial intelligence processing unit includes a Huawei Atlas200I A2 acceleration module, digital camera I, digital camera II, audible and visual alarm driver I, audible and visual alarm driver II, audible and visual alarm device I, audible and visual alarm device II, switching unit I, and power supply unit I. The acceleration module is connected to switching unit I, audible and visual alarm driver I, and audible and visual alarm driver II respectively. Switching unit I is connected to digital camera I and digital camera II. Audible and visual alarm driver I is connected to audible and visual alarm device I. Audible and visual alarm driver II is connected to audible and visual alarm device II. Power supply unit I provides power to acceleration module, switching unit I, audible and visual alarm driver I, and audible and visual alarm driver II.

[0027] The integrated wireless equipment for railway locomotives includes MMI human-machine interface terminal I, MMI human-machine interface terminal II, main control unit, satellite positioning unit, wireless communication unit, switching unit II, and power supply unit II. Switching unit II is connected to MMI human-machine interface terminal I, MMI human-machine interface terminal II, main control unit, satellite positioning unit, and wireless communication unit respectively. Power supply unit II provides power to each unit within the integrated wireless equipment for railway locomotives. The integrated wireless equipment for railway locomotives communicates with the artificial intelligence processing unit via network cable using the UDP protocol.

[0028] Railway Locomotive Integrated Wireless Equipment (CIR): Adopts locomotive wireless communication equipment conforming to TB / T 3375-2018 standard, integrating a satellite positioning unit (GPS / BeiDou dual-mode, positioning accuracy ≤10m), a wireless communication unit (4G / 5G+GSM-R dual-mode), a main control unit, and MMI human-machine interaction terminal I and MMI human-machine interaction terminal II. It is used to provide the system with accurate time, location, and speed data, realize data transmission between DMS and ground center, MMI human-machine interaction terminal I and MMI human-machine interaction terminal II, and display human-machine interaction alarm information.

[0029] The core of the Artificial Intelligence Processing Unit (DMS) adopts the Huawei Atlas 200I A2 acceleration module. The Atlas 200I A2 acceleration module integrates the Ascend 310 series AI processor, including one DaVinciV300 AI core with a main frequency of 0.5GHz and four TAISHANV200M processor cores with a main frequency of 1.0GHz. It integrates 4GB of RAM, achieving a computing power of 8 TOPS INT8. Internally, it has an independent hardware image encoding and decoding module, enabling various data analysis and inference calculations such as images and videos. The acceleration module provides a 314-pin MXM connector interface for expansion, supporting the connection of two digital cameras and an audible and visual alarm. Switching Unit I provides multiple network accesses; the audible and visual alarm driver I and driver II are connected to the Atlas 200I A2 acceleration module via digital I / O ports, driving and controlling audible and visual alarm I and II. The debugging network port provides system status information.

[0030] Digital cameras I and II are model SW-MI2018FX, 1080P@25fps, H.264 encoding, minimum illumination 0.05Lux, 12V power supply, support automatic ICR infrared filter switching, and have a lens focal length of 3.6mm. As front-end devices of the artificial intelligence processing unit, the digital cameras are connected to the DMS via a 100Mbps Ethernet interface, providing a stable RTSP video stream. Audible and visual alarm devices I and II are Guangzhou Yuexin Electronics YX95R-RGB, used to issue audible and visual alarms when the system detects abnormal driver behavior. The alarms integrate LED lights (supporting strobe, slow flash, and constant light) and buzzers (volume adjustable from 0-100db), connected to the DMS's audible and visual alarm driver via a wet junction box to ensure reliable alarm signal triggering.

[0031] Equipment Installation: To improve railway operating efficiency, the locomotive is equipped with two cabs. During train operation, the two cabs will not be used simultaneously. When the train arrives at its destination, the driver only needs to go to the other cab to take over and complete the train reversal. The MMI human-machine interaction terminal I, MMI human-machine interaction terminal II, audible and visual alarm I, audible and visual alarm II, digital camera I and digital camera II of the artificial intelligence processing unit (DMS) are respectively installed in the two cabs.

[0032] The Artificial Intelligence Processing Unit (DMS) is installed in a standard cabinet in the locomotive equipment room and connected to the communication interface of the Railway Locomotive Integrated Radio Equipment (CIR) in the same standard cabinet via a UDP network cable.

[0033] Digital camera I or digital camera II are fixed 1 to 1.5 meters horizontally in front of the driver's cab, and the angle is adjusted to ensure that the driver's seat area is completely covered.

[0034] The audible and visual alarm device I or the audible and visual alarm device II are respectively fixed above the dashboard diagonally in front of the driver's cab to ensure that the driver can see and hear it.

[0035] After the equipment is installed, confirm that the system is working properly. After the driver operates the MMI human-machine interface terminal I or MMI human-machine interface terminal II as the master control terminal in the cab on one side of the locomotive, the artificial intelligence processing unit will pull the video from the driver's digital camera I or digital camera II on the corresponding terminal side through the RTSP protocol for artificial intelligence analysis.

[0036] Prerequisites for determining working status: Based on real-time speed information from the Railway Locomotive Integrated Radio Equipment (CIR), when the speed is greater than 5 km / h, it is in working status and executes the process of processing the results of the artificial intelligence model algorithm; when the speed is less than or equal to 5 km / h, it only records and does not trigger an alarm (low-speed alarms can be enabled through system configuration for testing and demonstration).

[0037] Case 1, Absence Detection: The target detection process uses a model trained on YOLOv8x, which can effectively identify targets such as "person," "mobile phone," and "cigarette." The input pre-processed image size is 640×640; the confidence threshold for "person" is set to 0.7; the target detection algorithm determines in real time whether a "person" is detected in the center of the image (an area occupying more than 10% of the total image area centered on the driver's seat). If no person is detected for 10 consecutive seconds during operation, an absence is detected, and an "absence alarm" is added to the image's metadata and sent to the main process via inter-process communication; when a "person" is detected again in the central area of ​​the image, the absence detection timer is reset.

[0038] Assuming the locomotive is in operation (speed 80km / h), and the driver leaves his seat, such as... Figure 4 A schematic diagram of the detection box in object detection, showing the determination of detecting the target "person":

[0039] The image is a two-dimensional graphic, with the horizontal direction being X and the vertical direction being Y. The coordinates on the graphic are represented as (X,Y), with the top left vertex coordinates being (0,0) and the bottom right coordinates being (639,639). The image size is 640x640 (length x width). The output of the object detection model is: object type, top left corner coordinates of the bounding box, bottom right corner coordinates of the bounding box, and object confidence score.

[0040] For example, after the driver selects MMI human-machine interaction terminal I as the main control terminal, the main process will pull the video stream from digital camera I. After decoding and preprocessing (cropping, frame extraction, and adding metadata), the object data will be sent to the target detection process via inter-process communication. The target detection process will obtain the model input data from the object data as the input to the target detection algorithm. After calculation, it will output the target type "person", the upper left corner coordinates of the target box (100, 200), the lower right corner coordinates of the target box (470, 500), and the target confidence score: 0.85. Then, the target percentage = target area / image area = (470-100) * (500-200) / (640 * 640) = 26% > 10%, which is considered a valid target "person" detected. When the target confidence score is < 0.7 or the detection box for the target "person" is small, resulting in a target percentage < 10%, it is considered that no target "person" has been detected. Tstart is the system millisecond timestamp of the target detection process when the target "person" was last detected.

[0041] Step 1: If the target detection process detects a target "person", it updates Tstart to the current system number timestamp; if the target detection process does not detect a target "person", it does not need to update Tstart.

[0042] Step 2: Continuous detection. If no target "person" has been detected, determine the current system millisecond timestamp Tnow. When Tnow - Tstart >= 10 seconds, record "Away Alarm, First Detection Time 10:23:45" in the metadata and send it to the main process through inter-process communication. The main process controls the audible and visual alarm I on the same side to issue an audible and visual alarm. Update Tstart to the current system number timestamp and start the detection process again.

[0043] Case 2, Mobile Phone Detection: The target detection process uses a model trained on YOLOv8x, which can effectively identify targets "person," "mobile phone," and "cigarette." The input preprocessed image size is 640×640. In the detection algorithm, the confidence threshold for people is 0.7, and the confidence threshold for mobile phones is 0.8. The target detection algorithm determines whether both "person" and "mobile phone" targets are detected simultaneously. Furthermore, the detection box for the mobile phone target is located within the human target detection box (determined to be held by the driver). If the locomotive is in operation and detects more than 10 images meeting the conditions within 10 seconds, it is determined that a mobile phone is being used. The "mobile phone use alarm" type is appended to the image's metadata information and sent to the main process via inter-process communication; the count value of the mobile phone image is set to zero, and the mobile phone detection timer is restarted.

[0044] Assuming the locomotive is in operation (speed 80km / h), and the driver is talking on a mobile phone:

[0045] Step 1: After the driver selects the MMI human-machine interaction terminal I as the main control terminal, the main process will pull the video stream from digital camera I. After decoding and preprocessing (cropping, frame extraction, and adding metadata), the object data will be sent to the target detection process through inter-process communication. The target detection process will obtain the model input data from the object data as the input of the target detection algorithm. After calculation, the target detection process will continuously detect the "person" and "mobile phone" targets. The upper left coordinates of the "mobile phone" target are (180, 420), and the lower right coordinates are (200, 450). The upper left coordinates of the "person" target are (100, 300), and the lower right coordinates are (300, 639). The mobile phone frame is located within the human body frame, which satisfies the condition. The system's current millisecond timestamp for the first detection is recorded.

[0046] Step 2: Within 10 seconds (120 frames of images), a total of 12 images meeting the "human + mobile phone" condition are detected, exceeding the threshold of 10 images; the metadata records "mobile phone alarm, first detection time 10:23:45, a total of 12 valid images"; it is sent to the main process through inter-process communication, and the main process controls the audible and visual alarm I on the same side to issue an audible and visual alarm; the timestamp of the first detection is reset to the current system millisecond timestamp.

[0047] Case 3, Smoking Detection: The target detection process uses a model trained on YOLOv8x, which can effectively identify targets "person," "mobile phone," and "cigarette." The input preprocessed image size is 640×640. In the detection algorithm, the confidence threshold for people is 0.7, and the confidence threshold for cigarettes is 0.9. The target detection algorithm determines whether both "person" and "cigarette" targets are detected simultaneously. If more than 10 images of cigarettes are detected within 10 seconds, smoking is determined, a "smoking alarm" type is appended to the image's metadata information, and sent to the main process via inter-process communication; the cigarette target count is set to zero, and the cigarette detection timer is restarted.

[0048] Assuming the locomotive is in operation (speed 80km / h), and the driver is holding a cigarette:

[0049] Step 1: After the driver selects the MMI human-machine interaction terminal I as the main control terminal, the main process will pull the video stream from digital camera I. After decoding and preprocessing (cropping, frame extraction, and adding metadata), the object data will be sent to the target detection process through inter-process communication. The target detection process will obtain the model input data from the object data as the input to the target detection algorithm. After calculation, the target detection process will continuously detect the "human" and "cigarette" targets. The coordinates of the "cigarette" target are (180, 420) in the upper left and (185, 430) in the lower right, with a confidence level of 0.95. The coordinates of the "human" target are (100, 300) in the upper left and (300, 639) in the lower right, with a confidence level of 0.8. The cigarette is located within the human body frame and meets the confidence level requirement, so the condition is met. The system's millisecond timestamp of the first detection is recorded.

[0050] Step 2: Within 10 seconds (approximately 120 frames), a total of 11 images meeting the "person + cigarette" condition are detected, exceeding the threshold of 10 images; the metadata record "Smoking alarm, first detection time 12:34:56, total of 11 valid images" is recorded; the data is sent to the main process via inter-process communication, and the main process controls the audible and visual alarm I on the same side to issue an audible and visual alarm; the cigarette detection timer is restarted.

[0051] Case 4, Motion Detection: After the driver selects the MMI (Manual Machine Interface) terminal I as the main control terminal, the main process pulls the video stream from digital camera I. After decoding and preprocessing (cropping, frame extraction, and metadata addition), the object data is sent to the motion detection process via inter-process communication. The motion detection process obtains the original image data from the object data as input to the motion detection algorithm. The motion detection process uses the Image Perceptual Hash (pHash) algorithm, which is sensitive to the overall structure of the image and robust to changes in lighting and subtle variations. This algorithm generates a 64-byte image "fingerprint" hash value. Similarity is measured by calculating the Hamming distance (the number of different bits) between the hash values ​​of the current frame image and historical frame images; the smaller the distance, the more similar they are. If the similarity difference is large, exceeding a threshold, it is generally considered that the person in the image is not stationary. In the implementation, to detect fast and slow movement, two sliding window hash value sequences are set. The short window (0.5 seconds ago) calculates the Hamming distance between the current frame and the frame 0.5 seconds ago; if the distance is greater than 10, it is considered fast movement. The long window (3 seconds ago) calculates the Hamming distance between the current frame and the frame 3 seconds ago; if the distance is greater than 3 and less than 10, it is considered slow movement. If the Hamming distance between the two windows is less than 3, it is considered stationary. Metadata records "Motion Detection Short Window 15 Long Window 8" are sent to the main process via inter-process communication.

[0052] Case 5, Fatigue Driving Detection: The fatigue driving detection system integrates multiple detection technologies, including face detection, liveness detection, face marker localization, eye state analysis, and motion detection, forming a multi-dimensional fusion fatigue detection solution. After the driver selects the MMI human-machine interaction terminal I as the main control terminal, the main process pulls the video stream from digital camera I. After decoding and preprocessing (cropping, frame extraction, and adding metadata), the object data is sent to the fatigue detection process via inter-process communication. The fatigue detection process obtains the model input data from the object data as input to the "face" target detection algorithm for subsequent processing. 1. Face Detection: The fatigue detection process uses a pre-trained YOLOv8n-Face artificial intelligence model to detect "face" targets. The detection output includes: confidence score; upper left and lower right coordinates of the face; coordinates of five key points: eyes, nose, and corners of the mouth. Under the condition of meeting the confidence score, these are used as subsequent inputs. The pre-processed image size is 640×640. In the detection algorithm, the confidence threshold for the face is 0.7.

[0053] 2. Liveness Detection: To prevent drivers from using photos, printouts, or electronically displayed faces to circumvent fatigue detection, a liveness detection algorithm is introduced. The algorithm classifies real and fake faces based on the difference in high-frequency component distribution in the Fourier transform spectrum (fake faces have a more uniform high-frequency distribution, extending only horizontally or vertically, while real faces, due to stereo lighting, exhibit a center-to-outward divergence). The output is the "liveness" confidence score. In the detection algorithm: the input preprocessed image size is 128x128, and the liveness confidence threshold is 0.95.

[0054] Assume the driver uses a mobile phone screen to display a face-based adversarial detection: Step 1: The target detection process detects a "face" target; Step 2: The liveness detection algorithm scales the face region obtained in Step 1 to 128x128, performs a Fourier transform to obtain a spectrogram, and classifies the feature of the spectrogram to determine the confidence level of "liveness".

[0055] Step 3: When the classification is determined to be "deception", the image's metadata record is appended with "liveness detection alarm, detection time 10:45:12, face confidence 0.78, liveness confidence 0.82", and sent to the main process via inter-process communication; when the classification is determined to be "live", the following steps are executed.

[0056] 3. Face Marker Localization and Eye Opening / Closing: The fatigue detection process employs the PIPNet face marker localization algorithm to further analyze the detected facial information. This detection algorithm can locate and number 68 key points on the face, such as... Figure 3 As shown, specific organs can be selected by numbering, and this patent focuses on the position of the eye contour in the image.

[0057] Eye Aspect Ratio (EAR) is an important parameter used to measure the degree of eye closure. Calculation method: see appendix. Figure 3 Taking the left eye as an example, the corresponding marker points are numbered as p37 and p40 at the corner of the eye, p38 and p39 on the upper eyelid, and p41 and p42 on the lower eyelid. Each point is composed of its pixel position in the image.

[0058] Calculating Euclidean distance: Let there be two points p1(x1,x2) and p2(x2,y2) in a plane. The formula for the Euclidean distance between the two points is:

[0059] d(p1,p2)=sqrt((x1-x2) 2 +(y1-y2) 2 sqrt represents the square root operation.

[0060] Horizontal distance to the left eye: Lh = d(p37, p40);

[0061] The vertical distance to the left eye is 1: Lv1=d(p38,p42);

[0062] The vertical distance from the left eye is 2: Lv2=d(p3p19,p42);

[0063] Then the EAR of the left eye: LEAR=(Lv1+Lv2) / (2*Lh);

[0064] Similarly, the corresponding marked points for the right eye are p43 and p46 at the outer corner of the eye, p44 and p45 on the upper eyelid, and p47 and p48 on the lower eyelid. Calculate:

[0065] Horizontal distance of the right eye: Rh=d(p43,p46);

[0066] The vertical distance to the right eye is 1: Rv1=d(p44,p48);

[0067] The vertical distance to the right eye is 2: Rv2=d(p45,p47);

[0068] Then the EAR for the right eye is: REAR=(Rv1+Rv2) / (2*Rh);

[0069] Take the average value: EAR = (LEAR + REAR) / 2;

[0070] Set the threshold for EAR here to 0.2.

[0071] Assuming the locomotive is in operation (speed 80 km / h), the driver closes his eyes, falls asleep, and remains stationary:

[0072] Step 1: Facial landmark recognition process to extract facial landmarks of the driver:

[0073] The coordinates of the key points of the left eye are

[0074] p37:(220.0, 280.0);

[0075] p38:(230.0, 277.6);

[0076] p39:(240.0, 277.6);

[0077] p40:(250.0, 280.0);

[0078] p41:(240.0, 282.4);

[0079] p42:(230.0, 282.4);

[0080] Right eye key point coordinates

[0081] p43: (360.0, 280.0);

[0082] p44: (370.0, 277.45);

[0083] p45: (380.0, 277.45);

[0084] p46: (390.0, 280.0);

[0085] p47: (380.0, 282.55);

[0086] p48: (370.0, 282.55).

[0087] Step 2: Calculate the Euclidean distance:

[0088] Lh=d(p37,p40)=sqrt((220.0-250.0)²+(280.0-280.0)²)=30.0;

[0089] Lv1=d(p38,p42)=sqrt((230.0-230.0)²+(277.6-282.4)²)=4.8;

[0090] Lv2=d(p39,p41)=sqrt((240.0-240.0)²+(277.6-282.4)²]=4.8;

[0091] LEAR=(4.8+4.8) / (2×30)=0.16;

[0092] Step 3: Similarly, calculate the right eye REAR=0.17, EAR=(0.16+0.17) / 2=0.165<0.2, so it is determined that the eye is closed.

[0093] 4. Eye Opening / Closing Classification and Monitoring: The fatigue detection process employs a pre-trained AI model based on the YOLOv8n-Face classification task. The eye opening / closing classification detector is trained on a public dataset (Eyedataset containing over 100,000 close-up eye samples under different lighting and poses). This detector uses the eye contour location, determined by three face markers, as input to further assess the eye state. The output is the normalized confidence score for open and closed eyes, with a confidence threshold of 0.5. The results of this algorithm are appended to the metadata in the format "Fatigue driving detection, detection time 10:45:12, face confidence 0.78, liveness confidence 0.99, eye opening / closing degree 0.17, open / closed eye confidence 0.3", and sent to the main process via inter-process communication.

[0094] To ensure the accuracy of fatigue monitoring and avoid missed or false detections due to differences in facial features and environment, the main process employs three methods: a combination of eye opening / closing degree, eye opening / closing classification results, and motion detection Hamming distance values ​​for judgment.

[0095] The calculation of eye opening and closing degree is not strictly accurate due to individual differences, posture (squinting / head down / head tilting), and ambient lighting. In particular, it fluctuates around the 0.2 confidence level, leading to inaccurate judgment. Therefore, open and closed eye classification detection is introduced. Open and closed eye classification detection is also affected by posture and ambient lighting, and there is a probability of misjudgment.

[0096] Therefore, the results of both tests should be combined.

[0097] Step 1: When the eye opening / closing degree is <0.2 and the eye opening / closing classification test is "closed", it is identified as closed eyes;

[0098] Step 2: The motion detection process calculates that if the Hamming distance between the long window and the short window is ≤3, it is considered stationary;

[0099] Step 3: If the "eyes closed + stillness" condition is met for 10 consecutive seconds, the main program determines it as fatigue driving, triggers an alarm, and the main process controls the audible and visual alarm I on the same side to issue an audible and visual alarm; metadata is recorded in JSON format: Fatigue driving alarm, lasting 10 seconds, judgment dimensions: eyes + stillness. For situations where the eyes are open for an extended period and the person remains still, it is considered as focus and is not used as an alarm condition.

[0100] Test Results: For each anomaly detection scenario (leaving the seat, using a mobile phone, smoking, fatigue), anomaly scenario test videos were created through video editing. Each anomaly scenario video included 10 anomalies, each lasting 12 seconds. The anomaly scenario test videos were also used to communicate with the device via RTSP and played in a loop.

[0101] Through the above scenario tests, the present invention has an accuracy rate of ≥98% in identifying various abnormal scenarios under working conditions and an alarm response time of ≤300ms, which fully meets the actual needs of railway traffic safety monitoring.

[0102] like Figure 2 , Figure 3 As shown in the flowchart, the behavior monitoring method performs the following operations:

[0103] A1. Initialize and configure the hardware and software of the artificial intelligence processing unit, including creating the main process A and three processing processes: target monitoring process B, face detection process C, and motion detection process D. Main process A establishes a connection with the integrated wireless equipment for railway locomotives, including UDP connections with the satellite positioning unit and the main control unit. Once the connection is successful, the satellite positioning unit will periodically send its current location information, including time, latitude, longitude, and speed, to the main process at 1-second intervals. The main control unit will periodically send the status of the integrated wireless equipment for railway locomotives to the main process. After initializing parameters and loading the corresponding models, each processing process waits for the main process to send image information.

[0104] A2. Main process A communicates with the main control unit of the railway locomotive integrated wireless equipment via UDP, waiting for the main control unit to notify it to select the locomotive head or tail as the main control terminal. Once the driver operates the MMI human-machine interface terminal at the locomotive head or tail to perform check-in via buttons, the main control unit notifies main process A of the device address of the selected main control terminal.

[0105] A3. After obtaining the information that the front or rear of the vehicle is the main control terminal, the main process A pulls the data stream from the camera on the corresponding side through the standard RTSP protocol.

[0106] A4. The main process calls the internal image processing unit of the Huawei Atlas 200I A2 acceleration module to perform hardware decoding on the received RTSP stream, generating a static image array, typically 25 images per second. Considering the processing capability of the Huawei Atlas 200I A2 acceleration module and the fact that strict real-time processing is not required in actual use cases, frame-by-frame processing is performed on the images, i.e., processing is performed once for every other image. The processing includes: creating model input data: cropping and scaling the 1920x1080 image to 640x640, and normalizing it to meet the requirements of the input data format for subsequent model detection; adding metadata: including system millisecond timestamps, main control device identifiers, and location information of the satellite positioning unit of the railway locomotive integrated wireless equipment as metadata (JSON format); the object data consists of three parts: original image data, model input data, and metadata; the object data is sent to the receiving process (object detection process, face detection process, motion detection process) through inter-process communication mechanism.

[0107] B1. The object detection process subscribes to the message socket from the main process, obtains object data, and executes the object detection algorithm on the model input data to detect whether there are people, mobile phones, or cigarettes in the image.

[0108] B2. If an identifiable target is detected, the target's category, its position in the image (top left and bottom right coordinates), and the target's confidence level are encapsulated as detection results, added to the object data's metadata, and the object data is sent to the main process via inter-process communication.

[0109] C1. The face detection process subscribes to messages from the main process, obtains object data, executes the face detection algorithm on the model input data, and returns to continue waiting if no face is detected.

[0110] C2. If a face is detected, perform liveness detection, call the liveness detection model and algorithm; if no liveness is detected, add the detection result to the metadata of the object data and send it to the main process through the inter-process communication mechanism.

[0111] C3. If a live face is detected, the face tagging algorithm is invoked to generate a list of 68 marked points. If detection fails, the list will be empty.

[0112] C4. Based on the list of marked points obtained in C3, calculate the eye opening degree by referring to the index position. If the eye opening degree is less than the threshold value of 0.2, store it in the EAR list in the format of keyvalue, where the key is the system millisecond timestamp of the recording time and the value is the EAR value. Determine whether the key values ​​of the first and most recent entries in the list are greater than 10 seconds. If the condition is met, add it to the metadata of the object data.

[0113] C5. Based on the list of marker points obtained in C3, locate the coordinates of the corners of the eyes and eyelids of the left and right eyes, expand them into rectangular eye position boxes, input the position box coordinates into the eye opening and closing classification monitoring model, and calculate the classification results for the opening and closing detection of the left and right eyes respectively, with 0 for closed eyes and 1 for open eyes. Take the geometric mean of the results of the two eyes as the final result.

[0114] If the value is greater than or equal to 0.5, it is determined to be an open eye. The result is added to the object data's metadata. The object data is then sent to the main process via inter-process communication.

[0115] D1. The object detection process subscribes to the message socket from the main process, obtains object data, performs image-aware hashing (pHash) on the raw image data, and returns a 64-bit binary string;

[0116] D2. Store the results obtained from D1 (including the system millisecond timestamp and pHash value) into a sliding window queue, and calculate the Hamming distance with the pHash values ​​at the most recent 0.5 seconds and 3 seconds, respectively. If the Hamming distance is less than or equal to 3, the object is considered stationary. Add the results to the object data's metadata. Send the object data to the main process via inter-process communication.

[0117] A5. After the main process sends the object data to the receiving processing processes (object detection process, face detection process, motion detection process) via inter-process communication, it waits in a non-blocking manner for the processing result object data returned by other processes. Considering the different computation times of each process model, a maximum waiting timeout of 300ms is set (actual testing shows that results are generally returned within 150ms). After receiving the processing result object data returned by a processing process through inter-process communication, the main process A compares the system millisecond timestamp in its metadata with the current system millisecond timestamp. If it is greater than 300ms, the message is considered outdated and discarded. If it returns within 300ms, the following processing is performed: extract the metadata from the processing result object; check for alarms. If an alarm is detected: 1. A control signal is sent to the audible and visual alarm driver via the digital I / O port to trigger the strobe light (frequency 2Hz) and intermittent buzzer (1 second sound / 0.5 seconds pause, lasting 3 times); 2. The alarm information is sent to the CIR via the UDP protocol, and the CIR forwards it to the MMI human-machine interface terminal, displaying a specific alarm category icon in the upper right corner of the screen, accompanied by a buzzer reminder (0.5 seconds intermittent, lasting 3 times); 3. The current image and alarm information are packaged and uploaded to the ground center via the CIR's wireless communication unit. The ground center system displays an alarm pop-up and records the log.

[0118] The above process is for processing a single image. After processing a single image, return to step A3 and repeat the above steps to continue processing subsequent images.

Claims

1. A method for monitoring the driving behavior of railway drivers based on artificial intelligence, characterized in that, This is achieved using an artificial intelligence processing unit and integrated wireless equipment for railway locomotives, with the following steps: Step 1: After the artificial intelligence processing unit is powered on and running, it first completes the initialization of hardware interfaces, configuration of operating system environment, loading of logs and artificial intelligence algorithm models; Step 2: After the driver completes the main control terminal selection operation through the MMI human-machine interaction terminal at the front or rear of the train, the railway locomotive integrated wireless equipment sends the network IP address and device identification information of the main control terminal to the main process of the artificial intelligence processing unit in real time. Step 3: The main process of the artificial intelligence processing unit initiates a video stream retrieval request to the corresponding digital camera via the RTSP protocol based on the received network address of the master control terminal. Step four: After receiving the request, the digital camera continuously pushes the RTSP stream to the main process of the artificial intelligence processing unit through the data channel. After receiving the RTSP stream, the main process of the artificial intelligence processing unit decodes the stream into JPEG format images in real time through the acceleration module, crops and scales the images, and uses them as input for inference of various models; and adds complete metadata information to each image. Step 5: The three inference processes of the artificial intelligence processing unit run in parallel. Each process performs targeted inference and detection on the decoded JPEG image and metadata based on its own loaded model, and then transmits it to the main process. Step six: The main process receives and integrates the inference detection results from each inference process. For different abnormal behavior types, it makes a comprehensive judgment based on preset thresholds and the duration of abnormal states, thereby achieving triple alarms: local reminders, driver notifications, and remote monitoring, to regulate driver behavior.

2. The method for monitoring railway driver behavior based on artificial intelligence according to claim 1, characterized in that, The specific method for step one is as follows: initialize the hardware interface and initialize the software runtime environment; start the main process of the artificial intelligence processing unit acceleration module and three inference processes: target detection process, fatigue detection process, and motion detection process; The inference process initializes the resource management and memory management environment for the AI ​​runtime, and loads the corresponding trained AI models to support inference calculations for different categories of behaviors.

3. The method for monitoring railway driver behavior based on artificial intelligence according to claim 1, characterized in that, Step four is as follows: The main process receives the RTSP stream from the camera of the artificial intelligence processing unit and decodes it into a JPEG image through the acceleration module of the artificial intelligence processing unit; the JPEG image is preprocessed by cropping, scaling, normalizing, and frame extraction, and used as data input for the subsequent artificial intelligence model; metadata information is added to realize the association between the image and the train's status. The metadata information includes the millisecond-level timestamp of the local system, the standard time from the integrated wireless equipment of the railway locomotive, the precise latitude and longitude position, the speed of travel, and the identification of the main control device, to ensure the traceability of each frame of image. After the main process serializes the above three parts of data (decoding, preprocessing, and metadata addition), it publishes them to the inference process through the inter-process communication mechanism.

4. The method for monitoring railway driver behavior based on artificial intelligence according to claim 1, characterized in that, The specific method for step five is as follows: the target detection process, fatigue detection process, and motion detection process each perform the following artificial intelligence processing functions: the target detection process identifies people, faces, mobile phones, and cigarettes in the image; the fatigue detection process performs artificial intelligence processing such as face detection, liveness detection, face marker detection, and eye-open / closed classification. The motion detection process analyzes human movement; each detection process transmits the inference results, including target category, location, confidence level, key point coordinates, and status judgment results, to the main process in real time through inter-process communication mechanisms.

5. The method for monitoring railway driver behavior based on artificial intelligence according to claim 1, characterized in that, The specific method for step six is ​​as follows: The main process receives and integrates the inference detection results of each inference process, and makes a comprehensive judgment based on the preset threshold and the duration of the abnormal state to form alarm types such as absence alarm, mobile phone use alarm, smoking alarm, fatigue alarm, and liveness detection alarm. If the alarm conditions are met, the main program immediately triggers three types of responses: First, it controls the sound and light alarm device to issue a sound and light prompt through the digital IO port; second, it sends the alarm information to the MMI human-machine interaction terminal for display via the UDP protocol; and third, it reports alarm images and information to the ground center in real time via the wireless communication unit of the railway locomotive integrated wireless equipment through the UDP protocol.

Images