Locomotive driver operation fatigue evaluation system based on virtual reality

Abstract

The invention belongs to the technical field of locomotive driving man-machine work efficiency. The invention provides a locomotive driver operation fatigue evaluation system based on virtual reality. According to the embodiment of the invention, a high-simulation driving scene is constructed, a long-time and high-intensity driving task is simulated, various conditions possibly causing fatigue of the driver are reproduced in a risk-free manner, and the physiological reaction, behavior change and psychological state of the driver in the simulation process are monitored and analyzed in real time through high-precision equipment, so that the driving fatigue of the driver is improved. The problem that an existing evaluation means lacks accuracy and real-time performance is effectively solved, the operation fatigue degree of a locomotive driver in the task execution process can be accurately evaluated and found in time, and safety guarantee is provided for locomotive driving.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of locomotive driving ergonomics, and in particular to a locomotive driver fatigue evaluation system based on virtual reality. Background Art

[0002] During high-intensity and long-term driving operations, driver fatigue has become increasingly prominent and has become a major hidden danger affecting driving safety. Studies have shown that driver fatigue can lead to decreased reaction speed, distracted attention, weakened judgment, and other phenomena, which may lead to traffic accidents in severe cases. Especially in complex railway environments, the locomotive driver's ability to continue to work efficiently is a key factor in ensuring driving safety and maintaining transportation order. Therefore, in-depth research on the fatigue status of locomotive drivers is crucial to improving the safety and efficiency of railway transportation.

[0003] Although the hazards of driver fatigue have been widely recognized, the problems and shortcomings in current research are still prominent. Existing fatigue monitoring technologies often rely on physiological indicators. For example, some researchers use cameras and other sensors to monitor drivers' facial expressions, eye blinking frequency, head posture and other physiological indicators; some researchers use surface electromyography to collect electromyographic signals to achieve real-time monitoring and evaluation of work fatigue. These indicators are difficult to accurately capture in actual driving environments, and the evaluation has a certain degree of singularity. Although the currently commonly used methods can reveal the fatigue characteristics of drivers to a certain extent, they are limited by technical limitations and the diversity of fatigue representations in complex driving environments. These evaluation methods still need to be further improved in terms of accuracy and real-time performance, and have failed to help improve the driving safety of locomotives and reduce the possibility of accidents caused by driver fatigue. Moreover, the objective state of locomotive drivers during operation is difficult to be monitored by locomotive management personnel, especially when locomotive drivers are fatigued and driving accidents may occur. Therefore, the use of existing equipment and methods such as subjective scale evaluation and work fatigue evaluation based on physiological characteristics has certain defects in objectivity, comprehensiveness and accuracy. Summary of the invention

[0004] In order to avoid the shortcomings of the prior art, the present invention provides a locomotive driver work fatigue evaluation system based on virtual reality, which is used to solve the problem that the existing equipment and methods in the prior art, such as subjective scale evaluation and work fatigue evaluation based on physiological characteristics, have certain defects in objectivity, comprehensiveness and accuracy.

[0005] According to a first aspect of an embodiment of the present disclosure, a locomotive driver fatigue evaluation system based on virtual reality is provided, the system comprising: A virtual human-machine driving operation unit, used to provide simulated driving scenarios; A human-machine data comprehensive collection unit, used to collect the working behaviors and working tasks of the locomotive driver during driving to obtain driving working data; wherein the driving working data includes the working data, reaction time, accuracy, eye movement data and limb bone data of the locomotive driver; A data processing and feature extraction and analysis unit, used to pre-process the driving operation data and extract index feature values; wherein the index feature values ​​include workload information, task performance information, eye fatigue characteristics and limb fatigue characteristics; A cross-system data time synchronization and dimensionless processing unit for each indicator feature, used for time alignment of the operation data, and normalization and dimensionless processing of the indicator feature values ​​to obtain workload indicators, task performance indicators, eye movement feature indicators, and limb feature indicators; A fuzzy comprehensive evaluation unit for locomotive driver work fatigue, used for fusing the workload, the task performance, the eye movement characteristics and the limb characteristics to obtain a final evaluation result; The locomotive driver's operating fatigue comprehensive evaluation result grading unit is used to evaluate the locomotive driver's operating fatigue state according to the final evaluation result.

[0006] Furthermore, the virtual human-machine driving operation unit includes: Virtual car cockpit, virtual driving task, virtual reality head mounted display and motion capture device; among them, The virtual locomotive cockpit includes a seat, an operating console, a joystick, and a digital model of the locomotive cockpit, which is used to simulate the operating environment of the locomotive; The virtual driving tasks include a joystick control task, a driving situation gesture task, a road information observation task, and a dashboard information viewing task, which are used to simulate the driving task of a locomotive; The virtual reality head mounted display is used to track the eyes of the locomotive driver; The motion capture device is used to capture the driving motion of the locomotive driver.

[0007] Furthermore, the human-machine data comprehensive acquisition unit includes: Virtual car driving record module, driving task execution record module, eye movement data collection module and driving action collection module; among them, The virtual locomotive driving record module is used to collect the operation data of the locomotive driver; The driving task execution recording module is used to collect the reaction time and the accuracy rate of the locomotive driver during task execution; The eye movement data collection module is used to collect the eye movement data of the locomotive driver; The driving action collection module is used to collect the limb bone data of the locomotive driver.

[0008] Furthermore, the data processing and feature extraction and analysis unit includes: Workload calculation module, task performance calculation module, eye movement feature extraction module and limb feature extraction module; among them, The workload calculation module is used to calculate the operating behavior time and operating task requirement of the locomotive driver to obtain workload information; The task performance calculation module is used to obtain an average reaction time according to the reaction time and the accuracy, and to quantify the task performance information of the locomotive driver according to the average reaction time; The eye movement feature extraction module is used to preprocess the eye movement data and extract eye fatigue features from the preprocessed eye movement data; The limb feature extraction module is used to process the limb skeleton data and extract limb fatigue features from the processed limb skeleton data.

[0009] According to a second aspect of an embodiment of the present disclosure, a method for evaluating a locomotive driver's operating fatigue based on virtual reality is provided, the method comprising: Simulate driving scenarios; Based on the driving scenario, the working behaviors and working tasks of the locomotive driver during driving are collected to obtain driving working data; wherein the driving working data includes the working data, reaction time, accuracy, eye movement data and limb bone data of the locomotive driver; Preprocessing the driving operation data and extracting index characteristic values; wherein the index characteristic values ​​include workload information, task performance information, eye fatigue characteristics and limb fatigue characteristics; Performing time alignment on the operation data, and normalizing and dimensionlessizing the indicator characteristic values, so as to obtain a workload indicator, a task performance indicator, an eye movement characteristic indicator, and a limb characteristic indicator; fusing the workload, the task performance, the eye movement features, and the limb features to obtain a final evaluation result; The operating fatigue state of the locomotive driver is evaluated according to the final evaluation result.

[0010] Furthermore, the step of time-aligning the operation data and normalizing and dimensionlessly converting the indicator characteristic values ​​to obtain a workload indicator, a task performance indicator, an eye movement characteristic indicator, and a limb characteristic indicator includes: By calling the system time and the Unix timestamp, converting the Unix timestamp into a conventional date and time format; The cross-system data time synchronization and dimensionless processing unit for each indicator feature uses the extreme value normalization method to normalize and dimensionlessly process the workload information, the task performance information, the eye fatigue feature and the limb fatigue feature to obtain the workload index, the task performance index, the eye movement feature index and the limb feature index.

[0011] Furthermore, the workload index, the task performance index, the eye movement characteristic index and the limb characteristic index are secondary indicators; the tertiary indicators under the workload index are current workload and cumulative workload, the tertiary indicators under the task performance index are average reaction time, the tertiary indicators under the eye movement characteristic index are average pupil diameter and average blink time, and the tertiary indicators under the limb characteristic index are current discomfort and cumulative discomfort.

[0012] Furthermore, the step of fusing the workload, the task performance, the eye movement features and the limb features to obtain a final evaluation result includes: Construct membership function based on triangular fuzzy function; Performing confidence correction and consistency check on the judgment matrix of the membership function; The judgment matrix after confidence correction and consistency check is processed by using defuzzification calculation and root-finding method to obtain the maximum characteristic root and corresponding characteristic vector of the judgment matrix; Normalizing the maximum eigenvalue and the corresponding eigenvector to obtain the comprehensive weight of the secondary index; The secondary indicators are ranked in order of relative importance to the risk of driving fatigue, and combined with the comprehensive weights of the secondary indicators to determine the weight values ​​of all the third-level indicators; The weight values ​​of the calculated and dimensionless three-level indicators are weighted and summed with the corresponding weights to obtain a fatigue evaluation value, and the final evaluation result is determined according to the fatigue evaluation value.

[0013] Furthermore, the final evaluation results include: Not tired, slightly tired, somewhat tired, obviously tired and very tired; among them, The fatigue evaluation value of 0-0.2 indicates no fatigue; the fatigue evaluation value of 0.2-0.4 indicates slight fatigue; the fatigue evaluation value of 0.4-0.6 indicates relatively fatigue; the fatigue evaluation value of 0.6-0.8 indicates obvious fatigue; and the fatigue evaluation value of 0.8-1.0 indicates very fatigue.

[0014] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: In the embodiments of the present disclosure, through the above-mentioned locomotive driver fatigue evaluation system based on virtual reality, on the one hand, by constructing a highly simulated driving scene, simulating long-term, high-intensity driving tasks, reproducing various situations that may cause driver fatigue in a risk-free manner, and using high-precision equipment to monitor and analyze the driver's physiological reactions, behavioral changes and psychological states in the simulation process in real time, it effectively solves the problem of lack of accuracy and real-time performance of existing evaluation methods, and can timely and accurately evaluate and discover the degree of fatigue of locomotive drivers in the process of performing tasks, providing safety protection for locomotive driving. On the other hand, the system provides a more objective, comprehensive and scientific evaluation method for the evaluation of human-machine ergonomic locomotive driver fatigue, and promotes the innovative application of high-tech technologies such as virtual reality, motion capture and eye tracking in the field of traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0016] Figure 1 A schematic diagram showing a locomotive driver work fatigue evaluation system based on virtual reality in an exemplary embodiment of the present disclosure; Figure 2 A station operation logic flow chart designed in a virtual environment in an exemplary embodiment of the present disclosure is shown; Figure 3 A schematic top view of an experimental site constructed in a real environment based on the NOKOV optical motion capture system in an exemplary embodiment of the present disclosure is shown; Figure 4 A schematic diagram showing a process of converting motion-captured skeletal animation data into animation usable by SoErgo software in an exemplary embodiment of the present disclosure; Figure 5 A schematic diagram of a comprehensive evaluation index system for locomotive driver fatigue based on virtual reality in an exemplary embodiment of the present disclosure is shown; Figure 6 A driving fatigue value curve consistency comparison diagram in an exemplary embodiment of the present disclosure is shown; Figure 7 A graph showing the consistency comparison of the fatigue self-rating scale score curves of the subjects in the exemplary embodiment of the present disclosure is shown; Figure 8A diagram showing the steps of a method for evaluating a locomotive driver's fatigue based on virtual reality in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0018] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0019] In this example implementation, a locomotive driver fatigue evaluation system based on virtual reality is first provided. Figure 1 As shown in , the locomotive driver operating fatigue evaluation system based on virtual reality may include: a virtual human-machine driving operation unit, a human-machine data comprehensive acquisition unit, a data processing and feature extraction and analysis unit, a cross-system data time synchronization and dimensionless processing unit for each index feature, and a locomotive driver operating fatigue fuzzy comprehensive evaluation unit; Among them, the virtual human-machine driving operation unit is used to provide a simulated driving scene; the human-machine data comprehensive collection unit is used to collect the working behavior and working tasks of the locomotive driver during the driving process to obtain driving operation data; wherein the driving operation data includes the locomotive driver's operating data, reaction time, accuracy, eye movement data and limb bone data; the data processing and feature extraction and analysis unit is used to pre-process the driving operation data and extract indicator feature values; wherein the indicator feature values ​​include workload information, task performance information, eye fatigue characteristics and limb fatigue characteristics; cross-system data A time synchronization and dimensionless processing unit for each indicator feature is used to time align the operation data and normalize and dimensionless the indicator feature values ​​to obtain workload indicators, task performance indicators, eye movement feature indicators and limb feature indicators; a fuzzy comprehensive evaluation unit for locomotive driver operation fatigue is used to fuse the workload, the task performance, the eye movement features and the limb features to obtain a final evaluation result; a locomotive driver operation fatigue comprehensive evaluation result grading unit is used to evaluate the locomotive driver's operation fatigue state according to the final evaluation result.

[0020] Through the above-mentioned locomotive driver fatigue evaluation system based on virtual reality, on the one hand, by constructing a highly simulated driving scene, simulating long-term and high-intensity driving tasks, reproducing various situations that may cause driver fatigue in a risk-free manner, and using high-precision equipment to monitor and analyze the driver's physiological reactions, behavioral changes and psychological state in the simulation process in real time, it effectively solves the problem of lack of accuracy and real-time performance of existing evaluation methods, and can timely and accurately evaluate and discover the degree of fatigue of locomotive drivers in the process of performing tasks, providing safety protection for locomotive driving. On the other hand, the system provides a more objective, comprehensive and scientific evaluation method for the evaluation of human-machine ergonomic locomotive driver fatigue, and promotes the innovative application of high-tech technologies such as virtual reality, motion capture and eye tracking in the field of traffic safety.

[0021] Next, we will refer to Figures 1 to 7 Each part of the above-mentioned locomotive driver work fatigue evaluation system based on virtual reality in this example embodiment is described in more detail.

[0022] In one embodiment, the virtual human-machine driving operation unit provides a highly simulated experimental environment, the human-machine data comprehensive acquisition unit collects accurate objective data, and outputs it to the data processing and feature extraction analysis unit for data preprocessing and extracting effective feature values. The data time is unified and the data feature values ​​are dimensionless through the cross-system data time synchronization and each indicator feature dimensionless processing unit, and input into the locomotive driver's work fatigue fuzzy comprehensive evaluation unit to calculate and generate continuous and quantitative human-machine ergonomic work fatigue evaluation results. Finally, the results are input into the evaluation result grading unit to output the human-machine work fatigue status analysis results.

[0023] More specifically, the virtual human-machine driving operation unit includes: a virtual locomotive cockpit, a virtual driving task, a virtual reality head-mounted display, and a motion capture device. Among them: the virtual locomotive cockpit is a digital model of the locomotive cockpit established by digital modeling software, including seats, operating consoles, and joysticks; the virtual driving task simulates four typical tasks of locomotive driving, including the task of controlling the joystick, the task of driving gestures, the task of observing road information, and the task of viewing dashboard information; a high-end, high-precision virtual reality head-mounted display in the industry is used. The motion capture device includes a software and hardware system, in which the hardware mainly includes a POE switch, a POE separator, a calibration rod, an infrared optical lens, and a number of reflective identification points (marker points) with a diameter of 15mm.

[0024] The human-machine data comprehensive acquisition unit includes: a virtual locomotive driving record module, a driving task execution record module, an eye movement data acquisition module and a driving action acquisition module. Among them: the virtual locomotive driving record module collects the locomotive driver's operating data; the driving task execution record module collects the locomotive driver's reaction time and accuracy during task execution; the eye movement data acquisition module collects the locomotive driver's pupil size, blinking times and blinking time; the driving action acquisition module collects the locomotive driver's three-dimensional skeletal action data.

[0025] The data processing and feature extraction and analysis unit includes: workload calculation module, task performance calculation module, eye movement feature extraction module and limb feature extraction module. The workload calculation module includes the calculation of the locomotive driver's operating behavior time and the calculation of the operating task demand; the task performance calculation module quantifies the locomotive driver's task performance by the average reaction time obtained by reaction time and accuracy; the eye movement feature extraction module includes the preprocessing of eye movement data and the extraction of eye fatigue features; the driving action acquisition module includes the preprocessing of limb data patching, data processing based on animation redirection, and limb fatigue feature extraction based on SoErgo.

[0026] The fuzzy comprehensive evaluation unit of locomotive driver fatigue includes the following secondary indicators: workload indicator, task performance indicator, eye movement characteristic indicator and limb characteristic indicator. The third-level indicators under the workload indicator are current workload and cumulative workload; the third-level indicators under the task performance indicator are average reaction time; the third-level indicators under the eye movement characteristic indicator are average pupil diameter and average blink time; and the third-level indicators under the limb characteristic indicator are current discomfort and cumulative discomfort. All the third-level indicators will be dimensionless first.

[0027] The grading unit of the comprehensive evaluation results of locomotive driver's work fatigue includes five levels: the evaluation results between 0 and 0.2 are not tired; 0.2 to 0.4 are slightly tired; 0.4 to 0.6 are relatively tired; 0.6 to 0.8 are obviously tired; and 0.8 to 1.0 are very tired.

[0028] In a specific embodiment, Figure 1 As shown, a locomotive driver work fatigue evaluation system based on virtual reality of the present application includes: a virtual human-machine driving operation unit, a human-machine data comprehensive acquisition unit, a data processing and feature extraction and analysis unit, a locomotive driver work fatigue fuzzy comprehensive evaluation unit and a locomotive driver work fatigue comprehensive evaluation result grading unit.

[0029] The present application uses the virtual reality-based locomotive driver fatigue evaluation method of the above-mentioned system, including: driving records in driving tasks, task execution records, data preprocessing and feature extraction based on original eye movement and limb data, workload calculation, calculation and dimensionless conversion of three-level indicators, calculation of comprehensive evaluation results of locomotive driver fatigue based on virtual reality, and grading of evaluation results.

[0030] The virtual locomotive cockpit in the virtual human-machine driving operation unit uses Unity long-term support version 2022.3.13f1c1 as the construction platform of the virtual reality simulation environment to achieve a high degree of simulation of locomotive driving operation scenes. At the same time, the SteamVR plug-in version 2.2.3 is integrated, and the corresponding UnityPackage package is imported to ensure compatibility and interactivity with VR devices. The specific software and system used are shown in Table 1 Development environment.

[0031] Table 1 Development environment

[0032] The virtual environment created by the experiment includes the locomotive cockpit, platform, rails and road elements. The virtual locomotive cockpit is set to remain stationary, while the platform, rails and road are continuously and dynamically moved along the negative x-axis. A total of 10 prefabricated parts including a section of rails and the corresponding road surface are set in the scene, and each prefabricated part instance is mounted with the same script, using the "parallax scrolling" principle to synchronously adjust the position of the background elements. The key logic of the locomotive operation is as follows: Figure 2 As shown in the figure, collision detection points are precisely arranged on the platform. When the virtual train touches these collision points, the braking action will be triggered, the braking sound effect will be played, and the coroutine will be started to stay on the platform for 30 seconds.

[0033] The virtual driving task in the virtual human-machine driving operation unit is designed with four subtasks, namely, controlling the joystick, observing road information and giving feedback, viewing dashboard information and giving feedback, and making corresponding gestures. These tasks are mainly implemented through script development in Unity, where: the task rounds are called by using the coroutine function and the StartCoroutine() method in Unity to realize the function of starting, proceeding, and ending the task according to the time setting. The joystick control task is implemented through the Canvas canvas and its related functions in Unity. The road information observation task and the dashboard information viewing task are implemented by the interaction between the Unity virtual environment and the VR handle. Each round of the task lasts for 5 minutes, with a one-minute break after each round, for a total of ten rounds.

[0034] An eye tracking module integrated into the virtual reality head-mounted display in the virtual human-machine driving operation unit.

[0035] The motion capture system in the virtual human-machine driving operation unit uses an optical motion capture system. Figure 3 As shown, the overall size of the space is about 2 meters × 2 meters × 1.6 meters. The camera avoids blocking the subject, and there is no obvious noise in the space. Eight infrared cameras (numbered 1 to 8 in the figure) are evenly distributed around the experimental site to ensure that the subject's behavior and actions are covered from all angles. Located on both sides in front of the subject are two SteamVR space locators (numbered 9 and 10 in the figure). Their height positions are adjusted to be roughly the same as the VR handle and head display in the expected use, so as to minimize the drift of the handle and head display caused by positioning errors and ensure that the subject's actions can be accurately reflected in the virtual environment. In the core area directly in front of the subject, there is a bracket for the camera to shoot video (numbered 11 in the figure). Opposite to the video shooting bracket is a curtain set at the back (marked number 12 in the figure), which mainly plays the role of background isolation. Used to synchronously record the real-life video data of the experimental process. The main subject of the experiment (wearing a VR head display, numbered 13 in the figure) performs the preset task in the center of the venue, while the experiment host (numbered 14 in the figure) monitors and guides the entire experimental process outside the venue.

[0036] The virtual vehicle driving record module in the comprehensive human-machine data acquisition unit mainly records the operating behaviors and tasks during driving, providing a basis for the subsequent calculation of workload.

[0037] The driving task execution recording module in the human-machine data comprehensive acquisition unit obtains and records data into a CSV file through the Unity task script. Specifically, at each sampling moment, the system captures the current system time, the corresponding reaction time information, and the accuracy, and writes these data into a CSV file. The recording of all data is only activated during the execution of the task.

[0038] The eye movement data acquisition module in the human-computer data comprehensive acquisition unit captures eye movement data based on the virtual reality head-mounted display eye tracking module. The module can capture and record the timestamp, gaze origin, gaze direction, pupil position, pupil size and degree of eye opening during the task execution.

[0039] The driving action acquisition module in the human-machine data comprehensive acquisition unit is based on the Nokov series of passive optical motion capture systems with markers. According to the 53-point Helen Hayes model built into the software, marker reflective points are pasted on various joints of the human body. The three-dimensional coordinate information of these reflective points is captured by the system, and the driver's limb bone data can be calculated and generated.

[0040] The workload calculation module in the data processing and feature extraction analysis unit includes the calculation of the locomotive driver's operating behavior time and the calculation of the operating task demand. Among them: the operating behavior time includes the perception timeT v , cognitive time T c and operation time T m Three items.

[0041] First, information perception time T v Using the "Method-Time Measurement Method" (MTM) to estimate, according to the needs of the locomotive driver in this embodiment, the time for calculating information perception is about 4-11 TMU (146-396 ms), and the average is 7.3 TMU (263 ms).

[0042] Cognitive time for decision making T c Using Hick's law to estimate, the calculation formula is as follows:

[0043] In the formula, n is the number of optional items; The cognitive time of the selected item can be obtained from the statistical results of most people. The empirical value of is 155ms; For the i The frequency (or probability) of a choice occurring.

[0044] Use Fitts's law to calculate the execution time of an action T m The calculation formula is as follows:

[0045] In the formula, is the time for the operation to be executed, and its empirical value is 100ms; D is the manual distance required for the operation, and W is the diameter of the object being operated.

[0046] Therefore, the calculation method of the average operation time characteristic value of the operation behavior is:

[0047] If a certain task of the locomotive driver does not require any operation action, the time of this action is 0, that is, T v , T c and T m Can be 0.

[0048] The task demand also includes three items: perception, cognition and operation demand. The calculation formula is as follows:

[0049] In the formula, represents the required amount of perception task at that moment. Similarly, represents the cognitive task demand, is the operation task demand. B(t) is the duration symbol of the ith behavior element. When the behavior exists at time t, B(t)=1, otherwise B(t)=0. In this implementation, the task demand value range of each basic behavior element is adjusted to a 7-point system (the larger the score, the higher the corresponding task demand value). The specific values ​​of the task demand of each behavior element are shown in Table 2.

[0050] Table 2 Description of each visual, cognitive and operational basic behavior element and its task requirements Task Requirement Description of basic behavioral elements <![CDATA[ ρ v ]]> Visual Behavior 1 Visual Discovery 3.7 Visual Identity 4 Visual inspection 5 Visual Positioning 5.4 Visual Tracking 5.9 Visual Reading 7 Visual search, surveillance <![CDATA[ ρ c ]]> Cognitive Behavioral 1 Automatic response 1.2 Solution Selection 3.7 Signal Identification 4.6 Evaluation and judgment 5.3 Encoding and decoding <![CDATA[ ρ m ]]> Operational behavior 2.2 Discrete operations (buttons, trigger switches, etc.) 2.6 Continuous adjustment (steering wheel, speed handle, joystick, etc.) 4 Continuous hand operation 5.8 Discrete adjustments (rotation, vertical wheel, rod position) The average workload of each task is calculated as follows:

[0051] This application defines the above four subtasks (controlling the joystick, observing road information and giving feedback, viewing dashboard information and giving feedback and making corresponding gestures) as Task 1, Task 2, Task 3 and Task 4. Therefore, after calculating the operation behavior time and operation task requirement of each subtask respectively, the workload of this round of driving task can be calculated by the following formula:

[0052] represents the average workload of a task, where Indicates the time of a task behavior. The task performance calculation module in the data processing and feature extraction analysis unit selects the reaction time captured by the system and written into the CSV file in the experimental data and divides it by the accuracy rate, which is regarded as the average reaction time, and the average reaction time is used as the quantitative feature value of the task performance. Therefore, the calculation method of task performance is as follows:

[0053] In the formula, represents task performance, Represents the reaction time, Indicates the accuracy rate.

[0054] The eye movement feature extraction module in the data processing and feature extraction analysis unit is divided into two parts: data processing and feature extraction. In the eye movement data processing, first, the pupil diameters of the two eyes are selected. If the pupil diameters of both eyes are not -1, the average of the left and right eyes is calculated as the pupil diameter; if the data of one eye is -1, the data of the other eye is retained as the pupil diameter. Secondly, linear interpolation is used to fill in the missing pupil diameter data. Locate the missing data interval, select five consecutive valid data points before and after the interval, and use the average of the first five valid data points as the interpolation starting point, recorded as ; The mean of the last five valid data points is taken as the interpolation end point, denoted as . Fill in the missing data points in the middle and substitute the two points into the following formula for calculation:

[0055] In the formula, is the approximate function of this segment of data, is the original function of this segment of data, is the value of this point.

[0056] The pupil diameter data is eliminated with an FIR filter. This embodiment uses the scipy.signal library in Python to design and implement an FIR filter. By calling the firwin function and setting the corresponding filter order and the normalized cutoff frequency of 12.5 Hz, the filter coefficients that meet the requirements are systematically calculated. The pupil diameter data after interpolation is then low-pass filtered, and the high-frequency signal portion above 12.5 Hz is filtered out using a Hanning window.

[0057] In the eye movement feature extraction, two feature values ​​are mainly extracted, namely the mean pupil diameter (PDM) and the mean blink time (BTM). The mean pupil diameter (PDM) reflects the average size of the driver's pupil diameter, and its calculation formula is as follows:

[0058] Where N is the number of sampling points in the sample, is the pupil diameter. In order to test the difference of pupil diameter mean PDM under different fatigue levels of the same driver, the data of each subject were selected from the first five minutes and the last five minutes of the experiment, and the paired sample t-test was used for quantitative research. The significance level was set at 0.001. According to the t-test, it can be seen that there is a significant difference between the pupil data of the same driver under normal driving and fatigue driving.

[0059] If the pupil is less than 0.4 in several consecutive frames of data, this period is marked as a complete blink process. The duration of this period and the timestamp information of the starting frame are recorded. The average blink time (BTM) reflects the average time it takes for a driver to blink once, and its calculation formula is as follows:

[0060] Where N is the number of blinks, In order to test the difference of average blink time BTM under different fatigue levels of the same driver, the data of five minutes before and five minutes after the experiment were selected for each subject, and the paired sample t-test was used to quantitatively study the difference of average blink time. The significance level was set to 0.001, and it was found that there was a significant difference in the average blink time of the same driver under normal driving and fatigue driving.

[0061] The limb feature extraction module in the data processing and feature extraction analysis unit is divided into limb data repair preprocessing, data processing based on animation redirection, and limb fatigue feature extraction based on SoErgo. Limb data repair preprocessing is repaired through the motion capture system supporting software, and the interpolation is automatically calculated and the data is completed based on the data before and after the loss.

[0062] Data processing based on animation redirection requires exporting all data as FBX files, and then redirecting them in MotionBuilder. That is, through the skeleton model provided by MotionBuilder software, the animation data exported by the motion capture software is redirected to the animation data that SoErgo can use. The process is as follows: Figure 4 shown.

[0063] The limb fatigue feature extraction based on SoErgo uses the built-in RULA comfort analysis score as the feature value. Among them, the current time comfort is the feature value of the current discomfort, and the time-based cumulative comfort is the feature value of the cumulative discomfort.

[0064] In the cross-system data time synchronization and dimensionless processing unit for each indicator feature, because different hardware devices and software tools may use different methods to record data time information, by calling the system time and Unix timestamp, the Unix timestamp is converted into an easy-to-read date and time format to ensure that most data reaches millisecond-level time accuracy during the synchronization process, thereby effectively supporting experimental data analysis and integration. The integration formula is as follows:

[0065] Where AX represents the Unix timestamp value to be converted, and Date represents the time when the conversion is completed.

[0066] For the extracted indicator characteristic values, this implementation uses the extreme value normalization method to achieve the normalization and dimensionless of the quantitative indicator values. The specific operation steps are as follows: The positive indicators with larger values ​​and greater contributions to the target are:

[0067] The reverse indicators whose larger values ​​contribute less to the target are:

[0068] Where: represents the i-th observation value of the j-th indicator. and They represent the maximum and minimum values ​​that the jth indicator can take based on the upper and lower limits of the locomotive driver's working time and working conditions standards.

[0069] The fuzzy comprehensive evaluation unit for locomotive driver operational fatigue integrates dimensionless workload, task performance, eye movement characteristics and limb characteristics to obtain the final evaluation result. First, this implementation introduces the expert experience method of triangular fuzzy numbers to construct the membership function; secondly, the judgment matrix is ​​corrected for confidence and consistency test; then the defuzzification calculation and root-finding method are used to obtain the maximum eigenvalue and corresponding eigenvector of the judgment matrix. By normalizing the eigenvector finally obtained, the comprehensive weight of the indicator at this level can be obtained, and then the secondary indicators are sorted in order of relative importance to the driving fatigue risk, and the weight values ​​of all the third-level indicators are determined. Finally, the calculated and dimensionless eigenvalues ​​of the third-level indicators are weighted summed with the corresponding weights to obtain the final evaluation result. As Figure 5 As shown in the figure, it is a comprehensive evaluation index system of locomotive driver's work fatigue based on virtual reality.

[0070] The locomotive driver's fatigue comprehensive evaluation result grading unit divides the evaluation results into five levels, specifically: 0-0.2 is not fatigued; 0.2-0.4 is slightly fatigued; 0.4-0.6 is relatively fatigued; 0.6-0.8 is obviously fatigued; 0.8-1.0 is very fatigued. This grading can clearly analyze the fatigue status of the locomotive driver.

[0071] The system and method constructed in this embodiment have been further verified in the laboratory. Figure 6 and Figure 7 As shown, the driving fatigue value curve obtained in this embodiment and the fatigue self-rating scale score curve of the subjects show a consistent change pattern, and through correlation analysis, it is found that there is good consistency between the two.

[0072] Furthermore, in this exemplary embodiment, a method for evaluating the fatigue of a locomotive driver based on virtual reality is also provided. Figure 8 As shown in , the method includes: step S101 to step S105.

[0073] Step S101: simulating a driving scenario; Step S102: Based on the driving scenario, the working behaviors and working tasks of the locomotive driver during driving are collected to obtain driving working data; wherein the driving working data includes the working data, reaction time, accuracy, eye movement data and limb bone data of the locomotive driver; Step S103: pre-processing the driving operation data and extracting index characteristic values; wherein the index characteristic values ​​include workload information, task performance information, eye fatigue characteristics and limb fatigue characteristics; Step S104: performing time alignment on the operation data, and normalizing and dimensionlessizing the indicator characteristic values ​​to obtain a workload indicator, a task performance indicator, an eye movement characteristic indicator, and a limb characteristic indicator; Step S105: fusing the workload, the task performance, the eye movement features and the limb features to obtain a final evaluation result; Step S106: Evaluate the operating fatigue state of the locomotive driver according to the final evaluation result.

[0074] Specifically, the driving records in driving tasks, task execution records, data preprocessing and feature extraction based on original eye movement and limb data, workload calculation, calculation and dimensionless conversion of three-level indicators, calculation of comprehensive evaluation results of locomotive driver's work fatigue based on virtual reality, and grading of evaluation results.

[0075] The virtual locomotive cockpit in the virtual human-machine driving operation unit uses Unity long-term support version 2022.3.13f1c1 as the construction platform of the virtual reality simulation environment to achieve a high degree of simulation of locomotive driving operation scenes. At the same time, the SteamVR plug-in version 2.2.3 is integrated and the corresponding UnityPackage package is imported to ensure compatibility and interactivity with VR devices.

[0076] The virtual driving task in the virtual human-machine driving operation unit is implemented through script development in Unity, where: the task round is called by using the coroutine function and the StartCoroutine() method in Unity to realize the function of starting, proceeding, and ending the task according to the time setting. The joystick control task is implemented through the Canvas canvas and its related functions in Unity. The road information observation task and the dashboard information viewing task are implemented by the interaction between the Unity virtual environment and the VR handle. Each round of the task lasts for 5 minutes, with a one-minute break after each round, for a total of ten rounds.

[0077] An eye tracking module integrated into the virtual reality head-mounted display in the virtual human-machine driving operation unit.

[0078] The motion capture system in the virtual human-machine driving operation unit adopts an optical motion capture system.

[0079] The virtual vehicle driving record module in the comprehensive human-machine data acquisition unit mainly records the operating behaviors and tasks during driving, providing a basis for the subsequent calculation of workload.

[0080] The driving task execution recording module in the human-machine data comprehensive acquisition unit obtains and records data into a CSV file through the Unity task script. Specifically, at each sampling moment, the system captures the current system time, the corresponding reaction time information, and the accuracy, and writes these data into a CSV file. The recording of all data is only activated during the execution of the task.

[0081] The eye movement data acquisition module in the human-computer data comprehensive acquisition unit captures eye movement data based on the virtual reality head-mounted display eye tracking module. The module can capture and record the timestamp, gaze origin, gaze direction, pupil position, pupil size and degree of eye opening during the task execution.

[0082] The driving action acquisition module in the integrated human-machine data acquisition unit is based on the passive optical motion capture system with markers of the Nokov series. The principle of the system is: first, the marker points are attached to the driver according to the Helen Hayes model, and a group of high-performance lenses capture the spatial position information of each marker point by detecting the infrared light reflected back by the preset reflective marker points on its surface. The lens of this system is equipped with an infrared LED light source of a specific wavelength, which emits invisible infrared beams to the capture space; then, the high-reflectivity marker points attached to the object reflect these lights back to the lens, and the two-dimensional image information is captured by the high-speed image sensor integrated in the lens. After being processed by the image analysis algorithm, the coordinates of each marker point in the two-dimensional image will be resolved into three-dimensional space coordinates, and the corresponding motion parameters such as speed and acceleration will be further calculated. According to the Helen Hayes model built into the software, the driver's limb bone data can be calculated and generated.

[0083] The workload calculation module in the data processing and feature extraction analysis unit includes the calculation of the locomotive driver's operating behavior time and the calculation of the operating task demand. Among them: the operating behavior time includes three items: perception, cognition and operation time. The "Method-Time Measurement Method" (MTM) is used to estimate the information perception time. T v , using Hick's law to estimate the cognitive time of choice decisions T c , use Fitts's law to calculate the execution time of the operation T m Therefore, the calculation method of the average operation time characteristic value of the operation behavior is:

[0084] If a certain task of the locomotive driver does not require any operation action, the time of this action is 0, that is, T v , T c and T m Can be 0.

[0085] The task demand also includes three items: perception, cognition, and operation. This implementation adjusts the task demand value range of each basic behavioral element to a 7-point system (the larger the score, the higher the corresponding task demand value). The specific values ​​of the task demand of each behavioral element are shown in Table 1. The characteristic value of the average workload is the arithmetic square root of the sum of the squares of the three task demands. Therefore, the characteristic value of the current workload is obtained by multiplying the average operation time by the average workload. The characteristic value of the cumulative workload is obtained by accumulating the characteristic value of the current workload.

[0086] The task performance calculation module in the data processing and feature extraction and analysis unit selects the reaction time captured by the system and written into the CSV file in the experimental data and divides it by the accuracy rate, which is regarded as the average reaction time, and the average reaction time is used as the quantitative feature value of the task performance.

[0087] The eye movement feature extraction module in the data processing and feature extraction analysis unit is divided into two parts: data processing and feature extraction. In the eye movement data processing, first, the pupil diameters of the two eyes are selected. If the pupil diameters of both eyes are not -1, the mean of the left and right eyes is calculated as the pupil diameter; if the data of one eye is -1, the data of the other eye is retained as the pupil diameter. Secondly, the missing pupil diameter data is filled by linear interpolation. Finally, the FIR filter is used to eliminate the outliers and noise of the pupil diameter data. In the eye movement feature extraction, the sum of the pupil diameters of the sampling points within the sampling time is divided by the number of sampling points to obtain the average pupil diameter as the pupil diameter feature value. In the time series of pupil data, if the pupil is less than 0.4 in several consecutive frames of data, this period is marked as a complete blink process, and this period is the blink time. The sum of the blink time during the sampling period is calculated and divided by the number of blinks to obtain the value of the average blink time feature.

[0088] The limb feature extraction module in the data processing and feature extraction analysis unit is divided into limb data patching preprocessing, data processing based on animation redirection, and limb fatigue feature extraction based on SoErgo. Limb data patching preprocessing is patched by the supporting software of the motion capture system. According to the data before and after the loss, the interpolation is automatically calculated to complete the data. Data processing based on animation redirection requires exporting all data as FBX files, and then redirecting them in MotionBuilder, that is, through the skeleton model provided by MotionBuilder software, the animation data exported by the motion capture software is redirected to the animation data that SoErgo can use. The limb fatigue feature extraction based on SoErgo uses the built-in RULA comfort analysis score of the software as the feature value. Among them, the current time comfort is the feature value of the current discomfort, and the cumulative comfort based on time is the feature value of the cumulative discomfort.

[0089] In the cross-system data time synchronization and dimensionless processing unit for each indicator feature, because different hardware devices and software tools may use different methods to record data time information, by calling the system time and Unix timestamp, the Unix timestamp is converted into an easy-to-read date and time format, ensuring that most data reaches millisecond-level time accuracy during the synchronization process, thereby effectively supporting experimental data analysis and integration. For the extracted indicator feature values, this implementation uses the extreme value standardization method to achieve normalization and dimensionless numerical values ​​of quantitative indicators.

[0090] The fuzzy comprehensive evaluation unit for locomotive driver fatigue integrates dimensionless workload, task performance, eye movement characteristics and limb characteristics to obtain the final evaluation result. First, this implementation introduces the expert experience method of triangular fuzzy numbers to construct the membership function; secondly, the judgment matrix is ​​corrected for confidence and consistency test; then the maximum eigenvalue and corresponding eigenvector of the judgment matrix are obtained by defuzzification calculation and root-finding method. By normalizing the eigenvector finally obtained, the comprehensive weight of the index at this level can be obtained. Then, the secondary indicators are ranked according to the relative importance of the driving fatigue risk, and the weight values ​​of all the third-level indicators are determined. Finally, the calculated and dimensionless eigenvalues ​​of the third-level indicators are weighted summed with the corresponding weights to obtain the final evaluation result.

[0091] Compared with the existing methods, this embodiment provides a more objective evaluation method. All data are collected from equipment and systems and are not affected by the subjective consciousness of evaluators. The data collected by this embodiment is comprehensive, including body movement data that is difficult to measure and evaluate, which is more comprehensive and effective than the existing methods. This embodiment incorporates high-tech technologies such as virtual reality and motion capture systems, which can not only obtain high-precision objective data, but also various technologies bring convenience to the implementation process and greatly improve the evaluation efficiency of the experiment.

[0092] Through the above-mentioned locomotive driver fatigue evaluation system based on virtual reality, on the one hand, by constructing a highly simulated driving scene, simulating long-term and high-intensity driving tasks, reproducing various situations that may cause driver fatigue in a risk-free manner, and using high-precision equipment to monitor and analyze the driver's physiological reactions, behavioral changes and psychological state in the simulation process in real time, it effectively solves the problem of lack of accuracy and real-time performance of existing evaluation methods, and can timely and accurately evaluate and discover the degree of fatigue of locomotive drivers in the process of performing tasks, providing safety protection for locomotive driving. On the other hand, the system provides a more objective, comprehensive and scientific evaluation method for the evaluation of human-machine ergonomic locomotive driver fatigue, and promotes the innovative application of high-tech technologies such as virtual reality, motion capture and eye tracking in the field of traffic safety.

[0093] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0094] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A locomotive driver fatigue evaluation system based on virtual reality, It is characterized in that The system includes: A virtual human-machine driving operation unit, used to provide simulated driving scenarios; A human-machine data comprehensive collection unit, used to collect the working behaviors and working tasks of the locomotive driver during driving to obtain driving working data; wherein the driving working data includes the working data, reaction time, accuracy, eye movement data and limb bone data of the locomotive driver; A data processing and feature extraction and analysis unit, used to pre-process the driving operation data and extract index feature values; wherein the index feature values ​​include workload information, task performance information, eye fatigue characteristics and limb fatigue characteristics; A cross-system data time synchronization and dimensionless processing unit for each indicator feature, used for time alignment of the operation data, and normalization and dimensionless processing of the indicator feature values ​​to obtain workload indicators, task performance indicators, eye movement feature indicators, and limb feature indicators; A fuzzy comprehensive evaluation unit for locomotive driver work fatigue, used for fusing the workload, the task performance, the eye movement characteristics and the limb characteristics to obtain a final evaluation result; The locomotive driver's operating fatigue comprehensive evaluation result grading unit is used to evaluate the locomotive driver's operating fatigue state according to the final evaluation result.

2. According to the locomotive driver work fatigue evaluation system based on virtual reality in claim 1, It is characterized in that The virtual human-machine driving operation unit comprises: Virtual car cockpit, virtual driving task, virtual reality head mounted display and motion capture device; among them, The virtual locomotive cockpit includes a seat, an operating console, a joystick, and a digital model of the locomotive cockpit, which is used to simulate the operating environment of the locomotive; The virtual driving tasks include a joystick control task, a driving situation gesture task, a road information observation task, and a dashboard information viewing task, which are used to simulate a locomotive driving task; The virtual reality head mounted display is used to track the eyes of the locomotive driver; The motion capture device is used to capture the driving motion of the locomotive driver.

3. According to the locomotive driver fatigue evaluation system based on virtual reality in claim 2, It is characterized in that The human-machine data comprehensive acquisition unit comprises: Virtual car driving record module, driving task execution record module, eye movement data collection module and driving action collection module; among them, The virtual locomotive driving record module is used to collect the operation data of the locomotive driver; The driving task execution recording module is used to collect the reaction time and the accuracy rate of the locomotive driver during task execution; The eye movement data collection module is used to collect the eye movement data of the locomotive driver; The driving action collection module is used to collect the limb bone data of the locomotive driver.

4. According to the locomotive driver work fatigue evaluation system based on virtual reality in claim 3, It is characterized in that The data processing and feature extraction analysis unit comprises: Workload calculation module, task performance calculation module, eye movement feature extraction module and limb feature extraction module; among them, The workload calculation module is used to calculate the operating behavior time and operating task requirement of the locomotive driver to obtain workload information; The task performance calculation module is used to obtain an average reaction time according to the reaction time and the accuracy, and to quantify the task performance information of the locomotive driver according to the average reaction time; The eye movement feature extraction module is used to preprocess the eye movement data and extract eye fatigue features from the preprocessed eye movement data; The limb feature extraction module is used to process the limb skeleton data and extract limb fatigue features from the processed limb skeleton data.

5. A locomotive driver fatigue evaluation method based on virtual reality, It is characterized in that The method includes: Simulate driving scenarios; Based on the driving scenario, the working behaviors and working tasks of the locomotive driver during driving are collected to obtain driving working data; wherein the driving working data includes the working data, reaction time, accuracy, eye movement data and limb bone data of the locomotive driver; Preprocessing the driving operation data and extracting index characteristic values; wherein the index characteristic values ​​include workload information, task performance information, eye fatigue characteristics and limb fatigue characteristics; Performing time alignment on the operation data, and normalizing and dimensionlessizing the indicator characteristic values, so as to obtain a workload indicator, a task performance indicator, an eye movement characteristic indicator, and a limb characteristic indicator; fusing the workload, the task performance, the eye movement features, and the limb features to obtain a final evaluation result; The operating fatigue state of the locomotive driver is evaluated according to the final evaluation result.

6. According to the method for evaluating locomotive driver's fatigue based on virtual reality according to claim 5, It is characterized in that The step of time-aligning the operation data and normalizing and dimensionlessly converting the indicator characteristic values ​​to obtain a workload indicator, a task performance indicator, an eye movement characteristic indicator, and a limb characteristic indicator includes: By calling the system time and the Unix timestamp, converting the Unix timestamp into a conventional date and time format; The cross-system data time synchronization and dimensionless processing unit for each indicator feature uses the extreme value normalization method to normalize and dimensionlessly process the workload information, the task performance information, the eye fatigue feature and the limb fatigue feature to obtain the workload index, the task performance index, the eye movement feature index and the limb feature index.

7. According to the method for evaluating locomotive driver's fatigue based on virtual reality according to claim 6, It is characterized in that The workload index, the task performance index, the eye movement characteristic index and the limb characteristic index are secondary indicators; the tertiary indicators under the workload index are the current workload and the cumulative workload, the tertiary indicators under the task performance index are the average reaction time, the tertiary indicators under the eye movement characteristic index are the average pupil diameter and the average blinking time, and the tertiary indicators under the limb characteristic index are the current discomfort and the cumulative discomfort.

8. According to the method for evaluating locomotive driver's fatigue based on virtual reality as claimed in claim 7, It is characterized in that The step of fusing the workload, the task performance, the eye movement features and the limb features to obtain a final evaluation result includes: Construct membership function based on triangular fuzzy function; Performing confidence correction and consistency check on the judgment matrix of the membership function; The judgment matrix after confidence correction and consistency check is processed by using defuzzification calculation and root-finding method to obtain the maximum characteristic root and corresponding characteristic vector of the judgment matrix; Normalizing the maximum eigenvalue and the corresponding eigenvector to obtain the comprehensive weight of the secondary index; The secondary indicators are ranked in order of relative importance to the risk of driving fatigue, and combined with the comprehensive weights of the secondary indicators to determine the weight values ​​of all the third-level indicators; The weight values ​​of the calculated and dimensionless three-level indicators are weighted and summed with the corresponding weights to obtain a fatigue evaluation value, and the final evaluation result is determined according to the fatigue evaluation value.

9. The method for evaluating locomotive driver's fatigue based on virtual reality according to claim 8, It is characterized in that The final evaluation results include: Not tired, slightly tired, somewhat tired, obviously tired and very tired; among them, The fatigue evaluation value of 0-0.2 indicates no fatigue; the fatigue evaluation value of 0.2-0.4 indicates slight fatigue; the fatigue evaluation value of 0.4-0.6 indicates relatively fatigue; the fatigue evaluation value of 0.6-0.8 indicates obvious fatigue; and the fatigue evaluation value of 0.8-1.0 indicates very fatigue.

Images