Performance Measurement System and Method for Temporal Segregation of Eye, Physical, and Verbal Responses to Hazard Perception in Safety-Critical Job Tasks

Abstract

The invention relates to a system and method for measuring the stages of hazard perception and response in safety-critical work tasks. Hazard is defined as H(x,y,t); presentation start T0, first valid glance at the hazard area T1, perceptual acquisition T2 satisfying the duration / fixation and eye-data quality condition, independent physical response T3, and verbal response start T4 are recorded under the same trial ID and common time base. The system generates separate measurement fields such as Ls=T1-T0, Lr=T3-T2, and La=T4-T2. The raw event and data channels to which each measurement field is connected are recorded. If a channel does not meet the quality condition, only the measurements dependent on that channel are invalidated; for example, if the eye channel is invalid, T0 is preserved, and if the independent T3 is valid, Lt=T3-T0 is preserved. The synchronization internal consistency can be checked by directly comparing the sum of the event components with the total delay. Valid measurements can be transferred to optional statistical or artificial intelligence analysis and source-identified reporting. Proposed figure: Figure 1.

Description

1 TARIFF SAFETY-CRITICAL JOB TASKS AND HAZARD EVENTS THAT IMPACT EYE, PHYSICAL AND... VERBAL RESPONSES MEASUREMENT-DOMAIN-BASED VALIDITY CHECKED SYSTEM AND METHOD FOR TEMPORAL DECOMPOSITION Technical Field to Which the Invention Relates5 The invention relates to occupational health and safety, human-machine interaction, eye tracking, and task management. performance measurement, audio processing, time synchronization, and multimode data processing. It relates to these areas. The invention is particularly relevant for predefined safety-critical work tasks. or the temporal-spatial nature of a hazard event created by the stage engine ground-truth analyzes the user's eye, independent physical response, and verbal response events.10 matching under the same trial ID; which raw events each derived measurement field corresponds to and keeps track of which data channels it depends on; when an invalid channel occurs invalidates measurement areas dependent on that channel and preserves other measurements. It relates to a system and a method. State of the Art15 Computerized cognitive / fitness-for-duty assessments, eye movement reaction measurement, comparison of hazard location with gaze track, audio and visual data are all very important. combined with multimodal artificial intelligence and based on sensor reliability. Gating is known separately. Therefore, the defense approach for the invention is 'eye tracking'. 'use', 'voice analysis', 'use of artificial intelligence' or 'job-specific test20' It is not based on general elements such as 'being elected'. CN119539581A selects tasks according to professional requirements and uses Eye-RT. It uses Hand-RT data together. EP2523839A2, time-dependent hazard. detected by comparing their positions with the driver's gaze track data and It identifies undetected hazards. CN120360545A, time-stamped audio and 25 Speech-response-delay and eye movement features from eye movement data It extracts and produces a cross-modal fusion and evaluation report. EP3929824B1 calculates sensor-based uncertainty and identifies unreliable sensor inputs. It teaches a multimodal fusion approach that suppresses gating. The distinctive technical aspect of the invention is that the known parts in question are fused together in a general fusion.30 Rather than grouping them under one umbrella, the T0-T5 event chain for a single security incident and The relationship H(x,y,t)-G(x,y,t) is explained using a measure-domain-based dependency / validity logic. It is the use of. Technical Problems That the Invention Aims to Solve • Total reaction time is the difference between the visual presence of the hazard and the perceptual reaction time.35 Inability to distinguish between motor / verbal response delay after acquisition. • A sensory disorder such as eye-tracking loss is actually an independent condition that is valid. unnecessarily invalidates the measurement of physical response. • Although the dynamic hazard zone changes over time, the static AOI (Air-On-Import) 2 using the approach. • Combining different modalities without a common trial and timeframe. because the sequence of events becomes unclear. • Technical traceability between the measured event and the reported finding in the AI ​​report. disappearance.5 Explanation of the Figures • Figure 1: System architecture. • Figure 2: Event chain T0-T5. • Figure 3: Dynamic H(x,y,t)-G(x,y,t) mapping. • Figure 4: Measurement-area based validity matrix.10 • Figure 5: Sound channel. • Figure 6: Evidence-based AI reporting. • Figure 7: Flow of scientific calculation. Explanation of Reference Symbols Used in Figures 10: Visual presentation unit.15 20: Unit of operation / timing. 30: Physical response unit. 40: RGB camera. 50: NIR eye tracking unit. 60: Incident log / reporting unit.20 70: Common time base and trial ID. 80: Microphone / sound unit. 90: Quality and validity engine. 100: AI / statistical analysis and reporting unit. Disclosure of the Invention25 Summary of the Invention For each trial, the risk zone H(x,y,t) and the actual presentation start date T0 are recorded. The first valid intersection of the trajectory G(x,y,t) and H(x,y,t) is defined as T1; minimum The separate perceptual acquisition event T2 satisfies the dwell / fixation and eye-data quality condition. The physical response is determined by the T330 input circuit, independent of the eye-tracking channel. Thus, the start of the verbal response is T4 and, if necessary, the end of the verbal response is T5. It is recorded. Derived metrics are stored along with raw event dependencies. For example, Ls=T1-T0, Lp=T2-T1, Lr=T3-T2, Lt=T3-T0 and La=T4-T2. When the eye canal is invalid. Ls, Lp, Lr, and La, which are dependent on T1 / T2, can be overridden; however, T0 and independent T335 If valid, Lt is preserved. Thus, the validity decision applies to the entire sensor or the test. It applies not to the entire area, but only to the derived measurement range. 3 Danger Ground-Truth and Gaze Mapping The danger zone H(x,y,t) is a time-dependent mask, polygon, in the screen coordinates. It can be defined as a bounding box or pixel cluster. The eye point G(x,y,t) is the same. It is transferred to the coordinate system via calibration transformation. In the dynamic scene, T1 This can be defined as the following event: 5 T1 = inf{ t >= T0 : G(t) ∈ H(t) and qg(t) >= qmin} Here, qg(t) is the quality / confidence value of the eye sample. T2 is the hazard after T1. It is the initial acquisition moment that provides sufficient time and quality in the field. Example application: T2 No fixed universal threshold is claimed for this; dwell / fixation time and qmin are used. The eye tracker is parametric according to the sampling rate, task, and validation study.10 In the construction safety eye-tracking literature, fixations are mostly approximate. It has been reported that it is processed within minimum durations of 100-200 ms; this information constitutes an obligatory patent. It is used not as a threshold, but as a scientific basis for application design. Area-Based Validity Which raw events and modalities does each derived measure j depend on?15 A dependency matrix D(j,m) is maintained to show that there is a quality for each modality m. Acceptance Qm is established. Measurement validity in an application: Vj = ∏m I[ D(j,m)=0 or Qm >= θm ] In this way, binary; in another application, minimum or weighted confidence. It can be calculated as a continuous value using the function. Here I[.] indicator20 It is a function. Thanks to this structure, the invalidity of a modality is only applicable to that modality. It spreads to dependent measurements. Example: Lt=T3-T0 refers only to the screen event timing and physical response channel. It is dependent. Even if eye tracking is lost, if T0 and T3 are valid, Lt is reported. In contrast, since Lr=T3-T2, and T2 is dependent on Lr, if the gas quality is insufficient, Lr25. It becomes invalid. Sound Channel The voice unit timestamps the verbal response requested from the user during the task. It receives. T4 and T5 can be determined with voice activity detection. The voice channel also includes speech. speed, pause rate, energy, fundamental frequency and automatic speech recognition reliability30 It can extract technical specifications such as task-specific semantic accuracy, the recognized answer. matching the defined hazard class or key concept set for the same trial It can be calculated accordingly. Voice characteristics may reflect depression, anxiety, personality disorders, or similar clinical / psychological conditions. It does not have to be used for direct diagnosis. The core of the invention35 The technical effect is to link the verbal response event to the same H(x,y,t) security event and T2 The goal is to differentiate post-operative verbal initiation delay from other delays. Evidence-Based Calculations The following calculations are derived directly from the raw events measured by the system. 4 Universal job suitability thresholds are not claimed; normative thresholds are only relevant for target jobs. It is created through validation performed within the group. Measurement | Formula | Interpretation Visual search delay | Ls = T1 - T0 | From hazard presentation to first valid AOI input that much time.5 Perceptual acquisition delay | Lp = T2 - T1 | Valid dwell / fixation from the first AOI input. Time until acquisition. Motor response delay | Lr = T3 - T2 | From valid acquisition to physical response duration. Total physical response | Lt = T3 - T0 | If T0 / T3 is valid even if Gaze is invalid10 It can be protected. Verbal initiation delay | La = T4 - T2 | From current acquisition to speech initiation that amount of time. Verbal response time | Lv = T5 - T4 | Time between the start and end of voice activity. Algebraic control of temporal decomposition:15 Lt = T3-T0 = (T1-T0) + (T2-T1) + (T3-T2) = Ls + Lp + Lr T4-T0 = (T1-T0) + (T2-T1) + (T4-T2) = Ls + Lp + La These equations serve as exact decomposition checks when the sequence of events is valid. available; calculated component sum directly tolerance with total delay If there is a discrepancy, it may be marked as a trial time synchronization error.20 Task-Level Statistical Summaries For reaction times from current trials, the median is preferable; extreme values ​​are avoided. MAD = median(|xi - median(x)|) and robust as a measure of dispersion. Sigma approximately 1.4826 x MAD is available. These values ​​are for diagnosis or job suitability. It is not a threshold; it summarizes the performance distribution in the same task.25 If danger present / non-present trials are used, hit rate H and false-alarm rate F are separate. It can be reported. If desired, the signal detection theory sensitivity d' = Φ^-1(H) - Φ^-1(F) It can be calculated for validation and research purposes. If H or F is 0 / 1. Finite sample correction as H=(hits+0.5) / (Nh+1), F=(false alarms+0.5) / (Ns+1) can be used. The system can use this metric alone for suitability decision.30 He doesn't use it that way. Robust standardization when a job-specific reference sample is sufficient. This can be done with zR=(x-medianref) / (1.4826 x MADref). If there is no reference sample, the raw sample can be used. Measurements and confidence intervals are reported; arbitrary norms are not created. Calculation Verification Mechanism35 Two independent control methods are provided for each trial in the application software: (1) Lt=T3-T0 and T4-T0 are calculated directly from the time differences; (2) components The delays are calculated separately, and the sums Ls+Lp+Lr and Ls+Lp+La are formed. Two ways... If the absolute difference between them exceeds the clock resolution and rounding tolerance, it will be a trial. It is marked as a synchronization error. Thus, the calculation is based not only on the formula, but also on...40 It also has internal consistency checks during runtime. AI Fusion and Reporting An AI or statistical model runs after the core measurement system. The model alone... Vj refers to the measurement areas or measurement areas with valid measurement weights. Model architecture is not a necessary element of invention. The output is the event pattern, performance. This could be a set of criteria, a confidence value, or a similar auxiliary assessment. The reporting module first requires a configured evidence package P={trial ID, H definition, You can create {T0-T5, measurement areas, Vj, quality values, task / risk label}. Natural language The report is generated from this package, and the trial / measurement IDs on which the findings are based are... It is preserved. This feature is not the main innovation claim, but rather aimed at preserving the origin of the measurement.10 It is the preferred method. Job-Specific Applications The task library includes working at height, electricity, forklift / construction equipment, crane / lifting, machinery / manufacturing, mining / confined areas, chemistry / process, safety screening and similar fields. It may include different H(x,y,t) definitions for safety-critical risk families. Task by job15 The choice alone is not the element of novelty; it is the event-timing and This is the application context of the measurement validity chain. Psychological and Industrial Psychology Modules Vigilance / PVT-like tasks, selective reaction, in separate software modules, Go / No-Go, working memory, divided attention, situational judgment, and work-sample20 The tasks are applicable. License and terms of use with appropriate self-reporting tools. screening for depressive symptoms, anxiety, stress, drowsiness, or psychosocial risk. This can be done. These results are kept separate from core patent measurement and are used alone. It is not used for clinical diagnosis or automated hiring / termination decisions. PVT literature on fatigue-related performance in short-term vigilance tests25 This suggests it may be sensitive to lapses; however, certain lapse thresholds or These norms are universally accepted as a system without validation in the target population. It is not transferred as a limit. Scientific Basis • Basner M, Rubinstein J. Fitness for Duty: A 3-Minute Version of the30 Psychomotor Vigilance Test Predicts Fatigue-Related Declines in Luggage-Screening Performance. J Occup Environ Med. 2011;53(10):1146-1154. • Cheng B, Luo X, Mei X, Chen H, Huang J. A Systematic Review of Eye-Tracking Studies of Construction Safety. Front Neurosci. 2022;16:891725.35 • Jeelani I, Albert A, Han K, Azevedo R. Are Visual Search Patterns Predictive of Hazard Recognition Performance? J Constr Eng Manag. 2019;145(1):04018115. • Heng PP, Mohd Yusoff H, Hod R. Individual evaluation of fatigue at work to enhance the safety performance in the construction industry: A systematic 6 review. PLoS ONE. 2024;19(2):e0287892. • WHO. Guidelines on mental health at work / Mental health at work. 2022. • ILO. The psychosocial working environment: Global developments and pathways for action. 2026. Industrial Applicability5 The invention relates to occupational health and safety centers, vocational training and simulation centers, Human factors laboratories and safety-critical personnel assessment Their systems include a standard computer, screen, RGB / NIR camera, microphone, and This can be implemented with microcontroller-based physical response hardware.

Claims

7 REQUESTS 1. The user's response to a hazard event in a safety-critical job task. It is a system for measuring the stages of a hazard event in time and spatial terms. defining it as a region H(x,y,t) and the actual visual representation of the hazard event. The operation and timing unit (20),5 which records its start as a trial ID T0. generating the user's view trajectory G(x,y,t) and the first valid H(x,y,t) of G(x,y,t). The eye-tracking unit (50) that defines the intersection as T1, separate from T1, of the gaze A data quality condition on H(x,y,t) must be met with a minimum duration condition. The processing unit that identifies the time it provides as the T2 perceptual acquisition event, eye tracking. The physical response T3 is measured as time10 through an input circuit independent of the unit. stamping physical response unit (30), verbal response under the same trial identity The sound unit (80) that determines the beginning as T4 and the events T0, T1, T2, T3 and T4 It includes a synchronization unit (70) that keeps the system on a common time base; the system is the same For the trial, Ls=T1-T0 visual search latency, Lr=T3-T2 post-perceptual acquisition. physical response delay and verbal response after perceptual acquisition La=T4-T2115 It produces the start-up delay as separate measurement fields; each a specifying the raw events and data channels on which the derived measurement domain depends when it fails to maintain a measurement-dependency log and does not meet a data channel quality requirement only mark measurement areas dependent on the channel in question as invalid It is characterized by its preservation of other valid raw event and measurement areas.20 2. According to Claim 1, the system is in a trial where the eye-data quality condition is not met. T0 and eye tracking while marking measurement areas dependent on T1 and T2 as invalid. When T3 is valid independently of the channel, Lt = T3 - T0 is the total physical response. It is characterized by its ability to maintain its delay.

3. The system is according to claim 1 or 2, where H(x,y,t) is in the video or animation25 It is a dynamic hazard zone whose location, shape, or size changes over time. It is characterized.

4. Is the system, according to any of the previous requirements, and is the T2 perceptual acquisition event? minimum dwell time, fixation classification, gaze confidence, and gaze mapping. by creating it when at least two of the error conditions are met simultaneously.30 It is characterized.

5. The system is based on any of the previous requirements, and Lp=T2-T1 perceptual acquisition. additional measurement fields for delay and total physical response delay Lt=T3-T0 and the difference between Lt and Ls+Lp+Lr when the sequence of events is valid Synchronization is characterized by its calculation as an internal consistency check.35 6. The system is based on any of the previous requests, and the end of the verbal answer is T5. We define Lv as T5-T4 oral response time and T4-T0 as total oral response time. It generates the initial delay and T4-T0 when the event sequence is valid. The difference between Ls+Lp+La is considered as a synchronization internal consistency check. It is characterized by its calculation.40 7. The system is defined according to any of the previous requirements, and each derived measurement domain is j 8 The record D(j,m) shows the modality dependency for each modality m and the quality for each modality m. Its acceptance is characterized by establishing measurement-domain validity Vj using Qm. It is done.

8. The system is based on claim 7, where Vj has a trial ID within a validity bitmap and It is characterized by being stored along with the measurement area identifier.5 9. The system is based on any of the previous requirements, and the physical response unit is... being microcontroller-based and converting the physical input event from a user interface event It is characterized by independent timestamping.

10. The system is based on any of the previous requirements, and the voice unit is voice Activity detection to identify T4 and T5 and speech rate, pauses, basic10 at least one of the frequency, energy, or task-specific semantic accuracy characteristics It is characterized by its production.

11. The system, according to claim 10, automatically determines task-specific semantic accuracy. speech recognition output risk class or key associated with the same trial ID It is characterized by being determined by comparison with the concept set.15 12. The system is based on any of the previous requests and has the same trial ID. Using the presence and order of events H(x,y,t), G(x,y,t), T2, T3 and T4, at least visually no acquisition / no response, visual acquisition present / no physical response, delayed visual acquisition / fast physical response, early visual acquisition / delayed physical response, or response before T2. It is characterized by producing one of the event classes.20 13. The system is valid and the measurement is valid according to any of the previous requirements. fields such as median, median absolute deviation, time-incremental slope, hit rate, false-alarm rate or at least one of the derived task level summaries It is characterized by its production.

14. The system is valid according to any of the previous requirements, and the validity bitmap is valid25 Create a feature vector from the measurement areas marked as such and the relevant It is characterized by providing the vector to a statistical or artificial intelligence model.

15. The system according to claim 14, and the trial used by the model along with the model output. their IDs, measurement domain IDs, and validity values ​​are structured in the record. It is characterized by its ability to store information in this way.30 16. According to claim 15, the system must have at least one finding in the natural language report. by associating the trial ID and measurement domain ID on which it is based with the report record. It is characterized.

17. The system is based on any of the previous requirements and is suitable for different business / risk classes. It is characterized by its use of a task library containing separate H(x,y,t) hazard definitions.35 It is done.

18. The system is based on any of the previous requirements and displays RGB. Face visibility, head pose or looking off-screen obtained from unit (40). by using at least one of their pieces of information in establishing the eye-data quality condition is characterized.40 9 19. The user's response to a hazard event in a safety-critical job task. It is a method for measuring the stages of a hazard event, denoted as H(x,y,t). Defining and recording the actual presentation start as T0 with trial ID; G(x,y,t) Determine T1 as the first valid intersection of the viewing trajectory with H(x,y,t); from T1 separately, where both the time / fixation condition and the eye-data quality condition are met together.5 Determining the T2 perceptual acquisition event; eye-tracking channel-independent physical response. Timestamping as T3; starting oral response as T4 under the same trial ID. Specify as follows; keep T0-T4 events in a common time base; Ls=T1-T0, Generating Lr=T3-T2 and La=T4-T2 measurement areas separately; each measurement area generates the raw data. and recording channel dependency; and when a channel does not meet a quality condition10 only the measurement fields dependent on that channel are marked as invalid, while the other valid raw data... and is characterized by including steps to protect measurement areas.

20. The method is based on Claim 19 and is T1 / T2 based when the eye canal is invalid. If T0 and independent T3 are valid while invalidating the measurements, then Lt=T3-T0 measurement is preserved. It is characterized by including the step.15 21. The method is according to claim 19 or 20, and when the sequence of events is valid, Lt=T3-T0 The value is independently determined by the sum of Ls+Lp+Lr and the value of T4-T0 by the sum of Ls+Lp+La. Comparing them and the difference exceeding the defined time tolerance is called a synchronization error. It is characterized by being marked as such.

22. The method is according to any of claims 19-21, and is dynamic H(x,y,t)20 By mapping G(x,y,t) to time, events T1, T2, T3, and T4 can be mapped within the same trial. It is characterized by determining the order.

23. The method is according to any of claims 19-22, and each derived measurement domain Generating modality dependency records and validity values, and valid measurement. This includes the steps of providing the fields to artificial intelligence or statistical models.25 It is characterized.

Images