Railway driver distraction behavior identification method based on visual large model

Abstract

The invention discloses a railway driver distraction behavior identification method based on a visual large model, and belongs to the technical field of railway safety monitoring. Comprises: constructing a data set; based on the data set, training the visual big language model by adopting a LoRA method to obtain a fine-tuning visual big language model; obtaining human body key point information on the original video data frame, and generating a skeleton diagram; and inputting the skeleton graph into the fine-tuning visual large language model, and outputting description information of the image and a corresponding cue word. According to the method, the technical route of lightweight posture recognition and visual large language model analysis is fused, and the posture of the driver is captured by adopting a character key point detection tool; meanwhile, a visual large language model is introduced, and correlation analysis of operation behaviors and environment equipment is achieved by constructing a driving scene knowledge graph; the performance bottleneck of a traditional monitoring system is broken through, a technical template capable of being copied and popularized is formed, and the method has important practical significance for improving the intrinsic safety level of railway transportation.

Description

Technical Field

[0001] The present invention belongs to the technical field of railway safety monitoring, and in particular relates to a method for identifying railway driver distraction behavior based on a large visual model. Background Art

[0002] As the lifeblood of the national economy and a vital component of ensuring people's livelihoods, the safe operation of railway transportation directly impacts the safety of people's lives and property, as well as overall economic development. Train drivers, as the direct operators of the transportation system, have a crucial influence on train safety through their behavior. Over extended periods of driving, drivers can experience fatigue, drowsiness, inattention, or illegal operations, which can easily lead to major accidents such as train loss of control and derailment. This highlights the urgent need for intelligent behavior monitoring systems.

[0003] The current railway system generally adopts a supervisory model combining manual inspections with basic video surveillance. Traditional manual inspections suffer from blind spots and subjective judgment, making it difficult to achieve accurate, 24 / 7 monitoring. Existing video analysis systems, on the other hand, mostly use rudimentary, rule-based algorithms that can only recognize single actions like lowering the head or closing the eyes. Their accuracy is insufficient for combined movements, complex postures, and environmental interactions.

[0004] As in the prior art, a driver's line of sight mapping method based on a hybrid Transformer. The invention relates to a driver's line of sight mapping method based on a hybrid Transformer, and involves technical fields such as computer vision, image processing, and human-computer interaction. The driver's line of sight is closely related to the driver's attention and intention. In order to effectively estimate the driver's focus on the road environment during driving, the invention proposes a multi-scale cross-attention network. The network extracts the driver's facial features and eye features in parallel through Transformer and Resnet, and introduces cross-attention at different scales, which effectively solves the problem of decreased line of sight mapping performance due to redundancy of facial image features. In order to effectively fuse the driver's eye features, a binocular feature fusion module is proposed. This module fuses binocular features through the interaction of channel attention and spatial attention, significantly improving the line of sight mapping accuracy of MSCA-Net. The network can effectively estimate the driver's line of sight and can be widely used in fields such as driver attention analysis and human-vehicle interaction.

[0005] Its disadvantages are: (1) High computational complexity and insufficient real-time performance. Parallel Transformer and ResNet (especially hybrid structures) rely on a large number of self-attention mechanisms and multi-layer feature interactions. The computational complexity increases quadratically with the length of the input sequence (e.g., number of video frames, feature dimensions). In railway scenarios, continuous analysis of the driver's real-time video stream (e.g., above 30 fps) is required. High computational latency may result in delayed responses to distracting behaviors, failing to meet the stringent low-latency requirements of railway safety.

[0006] (2) Limited dependence on time sequence and weak generalization ability. This method focuses on the single behavioral feature of "line of sight", while railway driver distraction behaviors (such as fatigue driving, distraction, and operational errors) usually require a comprehensive judgment based on multimodal information (such as head posture, body movements, and operating equipment status). If the Transformer only uses line of sight data as input, it may miss complex distraction behaviors (such as looking down at a mobile phone without shifting the line of sight).

[0007] (3) Weak anti-interference ability and insufficient robustness. Line of sight occlusion problem: Drivers wearing sunglasses, lowering their heads to block their faces, or complex lighting (such as strong light in tunnels) may cause line of sight detection to fail.

[0008] Existing technology, a fatigue driving detection method based on a train cab scene. This invention belongs to the field of face detection technology, and specifically relates to a fatigue driving detection method based on a train cab scene. It first performs face detection using a deep learning method, detects 68 key points of the face, and uses the 68 points to calculate the distance between the eyes in the eye area, the distance between the lips, and uses the 68 points to convert to Euler angles to calculate the head direction and other three states to perform the first step of fatigue state judgment. After that, the visual features of the eye area are extracted to establish a visual model to judge whether the eyes are open or closed to improve the recognition ability of visual features. Then, the yawning judgment is performed in combination with the opening and closing features of the lips area, and the driver's fatigue is further judged based on the frequency of eye opening and closing and yawning, thereby reducing the misjudgment of a single feature and reducing the false alarm signals caused by misjudgment. Therefore, the scheme adopts a scheme combining two optimization strategies in terms of detection accuracy, which can accurately detect the fatigue state of the face.

[0009] Its disadvantages are: (1) Limited detection range: It only covers facial features and loses the semantics of whole-body behavior. It cannot capture non-facial behaviors. This solution relies entirely on facial key points (eyes, lips, head posture) to judge fatigue and cannot detect the railway driver's body movements.

[0010] (2) The single nature of fatigue detection is misaligned with railway safety requirements. In railway scenarios, non-fatigue distractions (such as attention loss) may account for a higher proportion, but this solution cannot cover them.

[0011] (3) Insufficient model scalability. Feature engineering relies on manual design and is difficult to adapt to new needs. This solution uses a fixed formula to calculate fatigue indicators (such as "blinking >20 times per minute is considered fatigue"), which is a heuristic rule and has the following problems: 1) The threshold is not generalizable: Different drivers have significantly different baseline blinking frequencies (e.g., novice drivers blink more frequently when nervous), and a fixed threshold is prone to false positives. 2) New distraction types are incompatible: Adding additional definitions requires redesigning facial features (such as the angle of the mouth corners and the frequency of eyebrow wrinkling), which increases development costs. The large-scale visual model automatically discovers key features (such as "hand close to the mouth + head tilted to the side") through end-to-end learning and supports zero-shot or few-shot transfer learning. Adding new distraction types only requires minor adjustments. Summary of the Invention

[0012] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art and provide a method for identifying railway driver distraction behaviors based on a large visual model, so as to solve the problem that the prior art has insufficient recognition accuracy for the driver's combined actions, complex postures and environmental interaction behaviors.

[0013] In order to achieve the above object, the technical solution adopted by the present invention is: A method for identifying railway driver distracted behavior based on a large visual model comprises the following steps: S1. Obtaining the original video data of the railway driver; S2, preprocessing the original video data to obtain a data set; S3. Based on the dataset, the LoRA method is used to train the visual language model to obtain a fine-tuned visual language model; S4, obtaining human body key point information on the original video data frame and generating a skeleton graph; S5. Input the skeleton image into the fine-tuned visual language model and output the image description information and the corresponding prompt words.

[0014] Furthermore, in S2, the original video data is pre-processed, including: The original video data is processed by frame extraction. The extracted images are segmented based on the region of interest to obtain the full body posture, facial expression, eye movement, and gesture areas in the image, and each area is marked with a different color; the extracted images are also subjected to distraction behavior label classification and risk rating.

[0015] Furthermore, the geometric sight lines of the eyes in the extracted frame image are calculated, including: Calculate the normalized ray corresponding to the pupil center pixel in the camera coordinate system: , ,

[0016] ,

[0017] Where, 、 、 Represents the normalized pixel coordinates; is the pixel coordinate of the pupil center; , is the main point coordinate; , is the focal length; Starting from the optical center, passing through the normalized plane point The ray represents the direction vector of the pixel point in the camera coordinate system; represents a unit vector; Based on the PnP algorithm, the rotation matrix of the head posture is obtained, and the normalized ray Perform the inverse rotation to obtain the unit direction vector of the pupil in the head coordinate system:

[0018] Where, is the unit direction vector in the head coordinate system; represents the head pose rotation matrix; 、 、 Represents the normalized head coordinates; Based on the unit direction vector, calculate the sight deflection angle:

[0019]

[0020] in:

[0021]

[0022] Where, is the horizontal yaw angle; is the vertical pitch angle; is the horizontal yaw angle in radians; is the vertical pitch angle in radians.

[0023] Furthermore, in S3, the LoRA method is used to train the visual language model, and the weight matrix in the Transformer layer in the visual language model is simplified to a low-rank counterpart, which specifically includes: Pre-trained weight matrix in Transformer layer , using the LoRA mechanism to introduce two low-rank matrices and ,get; ,

[0024] Where, 、 is the low-rank matrix learned during fine-tuning; is the fine-tuning matrix; d, k, and r represent the number of rows and columns of the matrix respectively; Fine-tuning the weights in the Transformer layer of the Vision-Large Language Model Expressed as:

[0025] The Transformer attention layer in fine-tuning the Vision-Large Language Model is represented as:

[0026] X Where, 、 and are the query matrix, key matrix and value matrix respectively, is the change of the bond matrix learned during fine-tuning; is the dimension of the key vector; is the visual feature of the input image.

[0027] Furthermore, the S4 specifically includes: Use human key point detection tools to obtain human key point information; On the original video frame, the key point information of each frame is connected with line segments and points to generate a skeleton graph.

[0028] Furthermore, the key point information includes facial key points, whole body posture key points, hand key points, driver's facial feature points, limb joint points, and outputs each frame Get the key point coordinates and confidence:

[0029] Where, Indicates the coordinates of key points; is the horizontal coordinate of the key point; List the coordinates of the key points, is the depth coordinate of the key point; is the number of frames; is the number of key points.

[0030] Furthermore, the S5 specifically includes: The skeleton graph input is broken down into multiple blocks in the fine-tuned visual language model, and the information in each block is understood, which is represented as:

[0031] Where, is a high-dimensional vector representing the visual features of a certain position or area in the skeleton image. It represents the jth Patch feature vector output by the visual encoder (ViT), which represents the visual information of a local area in the image; It represents the index of the patch, which is used to identify the location of the patch in the image; Indicates the total number of patches into which the image is divided; The output of the fine-tuned visual language model is represented as:

[0032] Where, Represents a visual large language model; Represents text input; is the text embedding function, is the visual embedding function; is the fine-tuned visual language model ontology, is the training parameter.

[0033] Furthermore, in S5, the skeleton image is input into the fine-tuned visual language model, and the regions in the skeleton image are identified according to the following method: A1. Facial status assessment: Perform an identification check of the driver's facial features, including eyes and mouth, to detect signs of driver distraction; A2. Hand-object interaction: Identify and observe the left and right hands, and determine the current hand movements and interaction objects, including operating instruments or other objects. A3. Dashboard interaction review: Shift attention to the left and right hands to determine interaction with the dashboard; A4. Behavioral analysis: Determine driver behavior classification based on facial state assessment, hand-object interaction, and dashboard interaction review; A5. Driving behavior risk assessment: Based on the visual cues and behavioral analysis results of the fine-tuned visual language model, the driver's distraction risk is assessed and the risk rating is determined.

[0034] The method for identifying railway driver distracted behavior based on a large visual model provided by the present invention has the following beneficial effects: The present invention integrates the technical route of lightweight posture recognition and visual large language model analysis, and adopts the human key point detection tool to capture the driver's posture; at the same time, it introduces the visual large language model and realizes the correlation analysis between operating behavior and environmental equipment by constructing a driving scene knowledge graph; it breaks through the performance bottleneck of the traditional monitoring system and forms a replicable and popularizable technical template, which has important practical significance for improving the inherent safety level of railway transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an example of image segmentation in an embodiment of the present invention.

[0036] Figure 2 Flowchart of a method for identifying railway driver distracted behavior based on a large visual model in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0038] The railway driver distracted behavior recognition method based on the visual large model in this embodiment integrates key point detection and temporal reasoning through the end-to-end large model to improve the accuracy and real-time performance of behavior recognition in complex scenarios. Figure 2 , which specifically includes the following methods: S1. Obtaining the original video data of the railway driver; S2, preprocessing the original video data to obtain a data set; In some embodiments, the original video data is subjected to frame extraction processing to obtain a frame-extracted image; in order to resolve irrelevant external environmental features, a feature focusing mechanism based on region of interest (ROI) segmentation is used to emphasize the driver's key features, including the driver's entire body, head, hands, and eyes, which are used to capture full-body posture, facial expressions, eye movements, and gestures, respectively. This targeted isolation directs the model's analysis to the driver's posture and movements, which is crucial for evaluating driving behavior. Figure 1 ,The color-coded boundaries in the body image (blue for the head, green for the right hand, red for the left hand, and orange for both eyes) can both highlight important regions and visually link these regions with the segmented images of the head, hands, and eyes.

[0039] In some embodiments, distraction behavior label classification and risk rating are performed on the extracted frame images, as shown in Table 1: Distracting behavior label classification and risk rating

[0040] Table 1 Distracting behavior categories and risk ratings Dataset construction; In some embodiments, a prompt template is created and used to request a targeted response from the visual big language model. By creating a dataset for fine-tuning the visual big language model and using the LoRA method to train the visual big language model, the model's ability to accurately classify and interpret driver behavior and related risk levels based on visual clues is enhanced, thereby obtaining a fine-tuned visual big language model.

[0041] Among them, for Figure 1 The facial region (blue) and eye region (orange) in the image are shown in Table 2 below:

[0042] Table 2 Visual input and prompt template In some embodiments, calculating the eye geometric sight line in the image obtained by frame extraction; Normalize pixel coordinates; assume that in the image coordinate system, the pixel coordinates of the pupil center (or iris center) are detected as , the camera intrinsic parameter matrix is:

[0043] in, , is the focal length (in pixels), , is the coordinate of the principal point; then the pixel point corresponds to the normalized ray in the camera coordinate system: , ,

[0044] make: ,

[0045] Head pose (rotation matrix): By using the PnP algorithm, several 3D face model points and the corresponding 2D image key points are input into OpenCV's solvePnP to obtain the rotation vector rvec and the translation vector tvec. The rotation vector rvec is converted into a rotation matrix:

[0046] Reverse projection to the head coordinate system: In order to obtain the direction vector of the pupil in the head coordinate system, it is necessary to normalize the ray Do the inverse rotation (ignoring translation), that is:

[0047] Here It is the unit direction vector in the head coordinate system (Head), approximately pointing to the direction of the tangent point between the center of the eyeball and the pupil plane.

[0048] Calculation of sight deflection angle: In the head coordinate system, the “head forward direction” is defined as a unit vector;

[0049] Will Decompose the coordinates in the coordinate system to obtain the horizontal (yaw) and vertical (pitch) deflection angles: Horizontal yaw angle ( ):

[0050] Vertical pitch angle ( ):

[0051] use Ensure that the specification of "head up" is consistent with the actual application (it can be reversed according to the system definition); Convert from radians to degrees, that is:

[0052] Where, 、 、 Represents the normalized pixel coordinates; It means starting from the optical center (the origin of the camera coordinate system) and passing through the normalized plane point The physical meaning of the ray is: the direction vector of the pixel point in the camera coordinate system (unnormalized length); Represents a unit vector, eliminating the influence of ray length and retaining only direction information; 、 、 Represents the normalized head coordinates; is the horizontal yaw angle; is the vertical pitch angle; is the horizontal yaw angle in radians; is the vertical pitch angle in radians.

[0053] S3. Based on the dataset, the LoRA method is used to train the visual language model to obtain a fine-tuned visual language model; In some embodiments, the LoRA method is used to train the visual large language model, and the weight matrix in the Transformer layer in the visual large language model is simplified to a low-rank counterpart, which specifically includes: Pre-trained weight matrix in Transformer layer , using the LoRA mechanism to introduce two low-rank matrices and ,get; ,

[0054] Where, 、 is the low-rank matrix learned during fine-tuning; is the fine-tuning matrix; d, k, r represent the number of rows and columns of the matrix respectively; the rank of these matrices is much smaller than The dimension of , makes the adaptive process parameters effective.

[0055] Fine-tuning the weights in the Transformer layer of the Vision-Large Language Model Expressed as:

[0056] This fine-tuning is applied to the attention and feed-forward layers of the Transformer, enabling the model to learn task-specific patterns while leveraging the extensive knowledge it has acquired during pre-training; the fine-tuning of the Transformer attention layer in the Vision-Large Language Model is represented as:

[0057] X Where, 、 and are the query matrix, key matrix and value matrix respectively, is the change of the bond matrix learned during fine-tuning; is the dimension of the key vector; is the visual feature of the input image.

[0058] The significance of LoRA fine-tuning is to capture the driver's distracted actions. The untrained visual language model has strong general capabilities, but does not focus on fine-grained actions such as "fatigue" and "making a phone call". LoRA update matrix What is learned is the direction of projection changes corresponding to these subtle movements, that is, in the original model space, a small number of directions strengthen or suppress attention responses, making the model more sensitive to distraction behavior characteristics.

[0059] S4, obtaining human body key point information on the original video data frame and generating a skeleton graph; In some embodiments, a human key point detection tool is used to detect facial key points (468 facial key points), body posture key points (33 body skeleton key points), and hands (21×2 key points) to extract the driver's facial feature points (such as eye opening, mouth corner curvature), limb joints (such as shoulders, elbows, hand positions), and hand key points (such as middle finger, index finger) in real time, and output each frame. Get the key point coordinates and confidence:

[0060] Where, Indicates the coordinates of key points; is the horizontal coordinate of the key point; List the coordinates of the key points, is the depth coordinate of the key point; is the number of frames; is the number of key points.

[0061] On the original video frame, the key point information of each frame is connected with line segments and points to generate a skeleton diagram, which "visualizes" the numerical information into the familiar RGB image format.

[0062] In a specific embodiment, taking the LLaVA1.5-7B visual language model as an example, LLaVA1.5-7B is generally composed of three modules and its structure is as follows: Vision Encoder: Converts an image (such as a skeleton image) into a set of "visual feature vectors".

[0063] Q-Former (Query Transformer): Extracts “key information” from visual features and converts it into a format that can communicate with the language model.

[0064] Language model (7B large language model, such as Vicuna-7B): understands image-related information + prompt words for classification, dialogue, or generation.

[0065] The skeleton image is fed into the ViT-Backbone (visual encoder) frame by frame or as a composite of multiple frames to obtain a patch-level feature sequence. This skeleton image is then treated as an image and fed into the pre-trained visual language model (ViT+Q-Former). This model breaks the image into small patches and understands the information in each patch.

[0066] Where, It is a high-dimensional vector representing the visual features of a certain position or area in the skeleton image; It represents the jth Patch feature vector output by the visual encoder (ViT), which represents the visual information of a local area in the image; It represents the index of the patch, which is used to identify the location of the patch in the image; Indicates the total number of patches the image is divided into; The output of the fine-tuned visual language model is represented as:

[0067] Where, Represents a visual large language model; Represents text input; is the text embedding function, is the visual embedding function; is the fine-tuned visual language model ontology, is the training parameter.

[0068] In the visual Transformer network, different attention weight mechanisms are used to assign higher attention weights to the head and hands, thereby improving the model's perception sensitivity to micro-movements in key areas (such as drowsiness and closing of eyes).

[0069] S5. Input the skeleton image into the fine-tuned visual language model and output the image description information and the corresponding prompt word; In some embodiments, the skeleton image is input into a fine-tuned visual language model, and regions in the skeleton image are identified using the following method: A1. Facial status assessment: Perform an identification check of the driver's facial features, including eyes and mouth, to detect signs of driver distraction; Specifically, the visual large language model examines the driver's facial features (blue rectangle), focusing on the eyes (orange rectangle) and mouth to detect signs of distraction, which may appear as yawning and possible gaze deviation.

[0070] A2. Hand-object interaction: Identify and observe the left and right hands, and determine the current hand movements and interaction objects, including operating instruments or other objects. Specifically, the left hand (green rectangle) and the right hand (red rectangle) are observed, where the visual large language model determines their involvement, whether it is operating an instrument or interacting with an object, to recognize actions such as smoking or eating, paying attention to the precise movements of the hands.

[0071] A3. Dashboard interaction review: Shift attention to the left and right hands to determine interaction with the dashboard; Specifically, at this stage, attention shifts to the left and right hands to determine the interaction with the instrument panel, which is crucial for identifying distractions caused by using railway instruments.

[0072] A4. Behavioral analysis: Determine driver behavior classification based on facial state assessment, hand-object interaction, and dashboard interaction review; Specifically, this step synthesizes the overall body language, resolves any remaining ambiguity, and clearly categorizes the driver's behavior. It carefully distinguishes between non-driving activities and appropriate driving controls.

[0073] A5. Driving behavior risk assessment: Based on the visual cues and behavioral analysis results of the fine-tuned visual language model, the driver's distraction risk is assessed and the risk rating is determined.

[0074] It also includes risk ratings, as shown in Table 1. Low risk levels (levels 1-3) trigger voice reminders, while high risk levels (≥level 4) trigger vehicle-mounted sound and light alarms, voice reminders ("Please stop and rest immediately") to send intervention signals.

[0075] In some embodiments, a system for identifying distracted railway drivers based on a visual macro model is provided, comprising: (1) Video acquisition module: vehicle-mounted wide-angle camera (resolution 1080P, frame rate 30FPS), covering the driver's entire body area.

[0076] (2) Data preprocessing module: Utilizing features based on region of interest (ROI) segmentation, a human key point detection tool is used to detect the face (468 key points), body (33 joints), and hands (21×2 key points) in real time; the key point sequence is normalized to generate a feature vector for input into the visual language model; and the driver's line of sight is geometrically estimated based on the eye ROI interception.

[0077] (3) Visual large model module: open source visual large language model + LoRA fine-tuning to obtain the fine-tuned visual large language model.

[0078] (4) Reasoning chain identification module: The reasoning chain outputs the exact category of distracting behavior and assigns risk ratings to different behaviors based on the reasoning results of the large model.

[0079] (5) Warning response module: Alert the driver according to the risk level (level 1-3 voice reminder, level 4-5 sound and light alarm).

[0080] Although the specific embodiments of the invention are described in detail in conjunction with the accompanying drawings, this should not be construed as limiting the scope of protection of this patent. Within the scope described by the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.

Claims

1. A method for identifying railway driver distracted behavior based on a large visual model, characterized by: The following steps are involved: S1. Obtaining the original video data of the railway driver; S2, preprocessing the original video data to obtain a data set; S3. Based on the dataset, the LoRA method is used to train the visual language model to obtain a fine-tuned visual language model; S4, obtaining human body key point information on the original video data frame and generating a skeleton graph; S5. Input the skeleton image into the fine-tuned visual language model and output the image description information and the corresponding prompt words.

2. The method for identifying railway driver distracted behavior based on a large visual model according to claim 1, characterized in that: In S2, the original video data is preprocessed, including: The original video data is processed by frame extraction. The extracted images are segmented based on the region of interest to obtain the full body posture, facial expression, eye movement, and gesture areas in the image, and each area is marked with a different color; the extracted images are also subjected to distraction behavior label classification and risk rating.

3. The method for identifying railway driver distracted behavior based on a large visual model according to claim 2, characterized in that: Calculate the eye geometry in the frame-drawn image, including: Calculate the normalized ray corresponding to the pupil center pixel in the camera coordinate system: , , , Where, 、 、 Represents the normalized pixel coordinates; is the pixel coordinate of the pupil center; , is the main point coordinate; , is the focal length; Starting from the optical center, passing through the normalized plane point The ray represents the direction vector of the pixel point in the camera coordinate system; represents a unit vector; Based on the PnP algorithm, the rotation matrix of the head posture is obtained, and the normalized ray Perform the inverse rotation to obtain the unit direction vector of the pupil in the head coordinate system: Where, is the unit direction vector in the head coordinate system; represents the head pose rotation matrix; 、 、 Represents the normalized head coordinates; Based on the unit direction vector, calculate the sight deflection angle: in: Where, is the horizontal yaw angle; is the vertical pitch angle; is the horizontal yaw angle in radians; is the vertical pitch angle in radians.

4. The method for identifying railway driver distracted behavior based on a visual large model according to claim 1, characterized in that: In S3, the LoRA method is used to train the visual language model, and the weight matrix in the Transformer layer in the visual language model is simplified to a low-rank counterpart, which specifically includes: Pre-trained weight matrix in Transformer layer , using the LoRA mechanism to introduce two low-rank matrices and ,get; ， Where, 、 is the low-rank matrix learned during fine-tuning; is the fine-tuning matrix; d, k, and r represent the number of rows and columns of the matrix respectively; Fine-tuning the weights in the Transformer layer of the Vision-Large Language Model Expressed as: The Transformer attention layer in fine-tuning the Vision-Large Language Model is represented as: X Where, 、 and are the query matrix, key matrix and value matrix respectively, is the change of the bond matrix learned during fine-tuning; is the dimension of the key vector; is the visual feature of the input image.

5. The method for identifying railway driver distracted behavior based on a visual large model according to claim 1, characterized in that: The S4 specifically includes: Use human key point detection tools to obtain human key point information; On the original video frame, the key point information of each frame is connected with line segments and points to generate a skeleton graph.

6. The method for identifying railway driver distracted behavior based on a large visual model according to claim 5, characterized in that: The key point information includes facial key points, whole body posture key points, hand key points, driver's facial feature points, limb joint points, and outputs each frame Get the key point coordinates and confidence: Where, Indicates the coordinates of key points; is the horizontal coordinate of the key point; List the coordinates of the key points, is the depth coordinate of the key point; is the number of frames; is the number of key points.

7. The method for identifying railway driver distracted behavior based on a visual large model according to claim 4 is characterized in that: The S5 specifically includes: The skeleton graph input is broken down into multiple patches in the fine-tuned visual language model, and the information in each patch is understood, which is represented as follows: Where, It is a high-dimensional vector representing the visual features of a certain position or area in the skeleton image; Represents the j-th Patch feature vector output by the visual encoder (ViT), which represents the visual information of a local area in the image; Represents the index of the patch, which is used to identify the location of the patch in the image; Indicates the total number of patches the image is divided into; The output of the fine-tuned visual language model is represented as: Where, Represents a visual large language model; Represents text input; is the text embedding function, is the visual embedding function; is the fine-tuned visual language model ontology, is the training parameter.

8. The method for identifying railway driver distracted behavior based on a large visual model according to claim 2, characterized in that: In S5, the skeleton image is input into the fine-tuned visual language model, and the regions in the skeleton image are identified according to the following method: A1. Facial condition assessment; Performing identification checks on the driver's facial features, including eyes and mouth, to detect signs of driver distraction; A2, hand-object interaction; Identify and observe the left and right hands, and determine the current hand movements and interaction objects, including operating instruments or other objects; A3. Dashboard interaction review: Shift attention to the left and right hands to determine interaction with the dashboard; A4, behavioral analysis; Determine driver behavior classification based on facial state assessment, hand-object interaction, and dashboard interaction review; A5. Driving behavior risk assessment; Based on the visual cues and behavioral analysis results of the fine-tuned visual language model, the driver distraction risk is assessed and the risk rating is determined.

Images