A method and system for detecting operator fatigue for long-duration spaceflight missions

Abstract

The present application relates to the technical field of intelligent behavior recognition, and particularly relates to a long-haul task personnel operation fatigue detection method and system. The method comprises the following steps: obtaining personnel image data and personnel voice data; performing voice trigger event recognition according to the personnel voice data to obtain voice trigger event data; performing facial dynamic feature extraction according to the personnel image data to obtain facial dynamic feature data; performing cross-modal anchor embedding of the facial dynamic feature data on the voice trigger event data to obtain multi-modal event data; performing reaction delay calculation on the multi-modal event data to obtain event reaction data; performing attention level processing according to the event reaction data to obtain event attention data; and performing fatigue determination on the event attention data to obtain personnel fatigue data. The present application effectively captures the reaction delay and attention fluctuation of the operator by constructing multi-modal mapping processing of voice trigger and facial response, thereby improving the accuracy of fatigue state recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent behavior recognition, and in particular to a long-haul task personnel operation fatigue detection method and system. BACKGROUND

[0002] With the gradual increase of long-haul transportation, sea maintenance, polar cruise and other tasks, ship drivers and task control personnel are facing long-time, high-alert and high-risk operation requirements. In a closed navigation environment, personnel are in information processing, high alert, day and night shift work rhythm for a long time, which is prone to physiological and psychological state fluctuations such as operation fatigue and decreased attention, and in severe cases, it will lead to delayed response to instructions, misoperation and even safety accidents. How to quickly and effectively identify personnel operation fatigue has become a problem. SUMMARY

[0003] The present application relates to the technical field of intelligent behavior recognition, and in particular to a long-haul task personnel operation fatigue detection method and system.

[0004] The present application provides a long-haul task personnel operation fatigue detection method, comprising the following steps:

[0005] Step S1: obtaining personnel image data and personnel voice data;

[0006] Step S2: performing voice trigger event recognition according to the personnel voice data to obtain voice trigger event data; performing face dynamic feature extraction according to the personnel image data to obtain face dynamic feature data; and performing cross-modal anchor embedding of the face dynamic feature data on the voice trigger event data to obtain multi-modal event data;

[0007] Step S3: performing reaction delay calculation on the multi-modal event data to obtain event reaction data;

[0008] Step S4: performing attention level processing according to the event reaction data to obtain event attention data; and performing fatigue determination on the event attention data to obtain personnel fatigue data.

[0009] In the present application, deep joint analysis of voice behavior and face dynamic features can be realized, cross-modal anchor embedding between voice trigger events and image data is based on, multi-source reaction signals in operation response can be effectively captured, and detection accuracy of reaction delay and attention fluctuation can be improved. By introducing reaction trajectory modeling and dynamic trend analysis, not only the instantaneous reaction capability can be quantified, but also the potential fatigue evolution process can be identified, and scheduling errors caused by single index misjudgment can be avoided.

[0010] Preferably, the personnel voice data includes first personnel voice data and second personnel voice data, and step S1 specifically comprises:

[0011] The personnel voice is collected through the cabin audio equipment and the personal microphone, and first personnel voice data and second personnel voice data are obtained respectively;

[0012] The personnel image is collected through the cabin camera, and personnel image data is obtained.

[0013] In the application, the dual-channel voice collection mode of the cabin audio equipment and the personal microphone is adopted, the clear near-field voice data and the environment voice can be obtained simultaneously in the high-noise environment of long voyage, the redundant capture of the voice signal and the adaptive switching of the signal-to-noise ratio are realized, and the accuracy and the robustness of the subsequent voice trigger event recognition are effectively improved. Meanwhile, the personnel image data is continuously collected through the cabin camera, the non-invasive monitoring capability is provided, the integrity and the continuity of the facial dynamic features are ensured, and the high-quality, multi-modal and synchronous alignment data support is provided for constructing the cross-modal time sequence response model.

[0014] Preferably, the voice trigger event recognition is specifically:

[0015] The channel alignment is performed on the first personnel voice data and the second personnel voice data, and channel alignment data is obtained;

[0016] The cross-channel mutual verification recognition is performed on the channel alignment data, and cross-channel fusion data is obtained;

[0017] The double-layer keyword trigger is performed on the cross-channel fusion data, and trigger detection data is obtained;

[0018] The voice trigger event is generated according to the trigger detection data, and voice trigger event data is obtained.

[0019] In the application, the channel alignment and the cross-channel mutual verification are performed in the voice trigger event recognition process, the complementary characteristics between the first personnel voice data (the cabin channel) and the second personnel voice data (the personal channel) are fully utilized, the recognition accuracy and the fault tolerance are improved through the confidence comparison and the difference correction in the case of background noise or voice distortion, the double-layer keyword trigger strategy is adopted, the acoustic feature matching and the semantic text analysis are combined, the response sensitivity to the key instruction and the anti-mis-triggering capability are improved, the voice trigger event data formed by the system has the characteristics of high confidence, high synchronization and high matching, a stable entrance for behavior analysis is provided, and the overall intelligent discrimination level of the system is effectively enhanced.

[0020] Preferably, the cross-channel mutual verification recognition is specifically:

[0021] The single-channel voice recognition is performed on the channel alignment data, and voice recognition data is obtained;

[0022] The consistency comparison is performed according to the voice recognition data, and consistency comparison data is obtained;

[0023] According to the consistency comparison data, difference positioning is performed to obtain difference positioning data;

[0024] According to the difference positioning data, confidence weighting discrimination is performed to obtain weighted discrimination data;

[0025] According to the weighted discrimination data, re-discrimination is performed to obtain re-discrimination data;

[0026] The re-discrimination data is fused across channels to obtain cross-channel fusion data.

[0027] In the cross-channel mutual authentication recognition process in the application, by performing single-channel speech recognition on the double-channel speech data respectively, combining the consistency comparison and difference positioning strategy, the local conflict area of the two-way speech on the keyword and semantic paragraph can be accurately identified; through confidence weighting discrimination, the reliability degree of the recognition result in the difference area is dynamically adjusted, the discrimination robustness in the cabin under high noise and multiple speaker environment is improved; at the same time, the re-discrimination process is set, the reinforcement model is called for re-judgment for the low consistency degree section, and the fusion result output is more reliable in terms of text accuracy and semantic integrity, so as to improve the overall performance and system stability of the voice trigger event recognition.

[0028] Preferably, the double-layer keyword trigger specifically comprises:

[0029] Acoustic layer candidate detection is performed on the cross-channel fusion data to obtain acoustic layer candidate data;

[0030] Text layer keyword detection is performed on the cross-channel fusion data to obtain text layer keyword data;

[0031] Cross-layer consistency verification is performed on the acoustic layer candidate data and the text layer keyword data to obtain cross-layer consistency data;

[0032] Trigger confidence evaluation is performed on the cross-layer consistency data to obtain trigger detection data.

[0033] In the double-layer keyword trigger process in the application, acoustic layer candidate detection and text layer keyword recognition are innovatively fused to realize double-path capture of potential voice trigger intent. The acoustic layer is based on low-level features such as energy mutation and intonation profile of audio signals, which can realize preliminary screening of non-standard pronunciation, ambiguous sentences or accent variation instructions; and the text layer performs semantic-level keyword accurate matching through speech recognition results. Through cross-layer consistency verification, the false trigger and missed trigger phenomena are effectively eliminated, and reliable trigger detection data is generated combined with the confidence evaluation strategy, which enhances the stability and response accuracy of the voice trigger event recognition in the complex cabin communication environment.

[0034] Preferably, the face dynamic feature extraction specifically comprises:

[0035] Face detection is performed according to the personnel image data, and face detection data is obtained;

[0036] Facial key point recognition is performed on the face detection data, and facial key point data is obtained;

[0037] Eye feature extraction and mouth feature extraction are performed according to the facial key point data, and eye feature data and mouth feature data are obtained, respectively;

[0038] Cross-part association processing is performed according to the eye feature data and the mouth feature data, and cross-part association data is obtained;

[0039] The cross-part association data is subjected to collaborative mode mapping generation, and facial dynamic feature data is obtained.

[0040] In the present application, in the process of facial dynamic feature extraction, the personnel image data is first subjected to face detection and facial key point recognition, the eye and mouth regions are accurately positioned and structurally stable extracted, and the dynamic features are ensured to have time sequence continuity and local distinguishability. Cross-part association processing is proposed, the time sequence coupling relationship between eye movement (such as blinking, closing eyes) and mouth behavior (such as opening mouth, yawning) is analyzed, and high correlation fatigue representation mode is effectively mined. Multi-dimensional dynamic features are generated through collaborative mode mapping, which can reflect micro-expression changes and multi-part linkage trends, provide more accurate and fine-grained physiological behavior feature support for fatigue determination, and improve the perception depth and discrimination accuracy of the model for potential fatigue state.

[0041] Preferably, the cross-modal anchor embedding comprises:

[0042] Voice event anchor point extraction is performed on the voice trigger event data, and voice event anchor point data is obtained;

[0043] Face feature segment extraction is performed according to the voice event anchor point data, and face feature segment data is obtained;

[0044] Time sequence alignment is performed according to the voice event anchor point data and the face feature segment data, and time sequence alignment data is obtained;

[0045] Anchor point embedding is performed according to the time sequence alignment data, and multi-modal event data is obtained.

[0046] In the cross-modal anchor embedding process, the time anchor points in the speech trigger event are extracted, and the dynamic feature segments in the corresponding period of the face image sequence are combined to realize the accurate timing alignment between the speech and the face modal. Through the anchor point embedding mechanism, the face micro-expression changes and the speech response behavior are associated on a unified time axis to construct multi-modal event data with time consistency and semantic coupling. The asynchronous and modal split problems in traditional multi-modal processing are avoided, the context integrity and dynamic perception ability of reaction delay modeling and fatigue state recognition are improved, and the state fusion and behavior understanding in the human-computer interaction context in the long voyage environment are suitable.

[0047] Preferably, step S3 is specifically

[0048] The trigger anchor point time data is obtained by performing trigger anchor point time acquisition on the multi-modal event data.

[0049] The response behavior time data is obtained by performing response behavior time capture on the trigger anchor point time data.

[0050] The reaction time difference data is obtained by performing reaction time difference calculation on the response behavior time data.

[0051] The event reaction data is obtained by performing reaction feature extraction according to the reaction time difference data.

[0052] The reaction feature extraction is specifically:

[0053] The reaction trajectory data is obtained by performing reaction trajectory construction according to the reaction time difference data.

[0054] The reaction dimension data is obtained by performing phase space dimension embedding on the reaction trajectory data.

[0055] The double-domain construction data is obtained by performing trend fluctuation double-domain construction according to the reaction dimension data.

[0056] The trajectory geometry data is obtained by performing trajectory geometry quantization according to the double-domain construction data.

[0057] The event reaction data is obtained by performing change point detection on the trajectory geometry data.

[0058] In the present application, by performing trigger anchor point time acquisition and response behavior time capture operations on multi-modal event data, the timing mapping relationship between speech trigger and face behavior can be accurately established, and the calculation error caused by modal non-synchronization or delay offset can be effectively avoided. The reaction time difference calculation performed by the system not only supports high-precision response time extraction, but also provides original support for reaction trajectory analysis. Through reaction feature extraction, a rich dynamic response feature vector can be constructed to represent the operation agility and mental reaction level of individuals in a specific task context, and the explainability and judgment accuracy of fatigue recognition are improved.

[0059] This invention, in the process of reaction feature extraction, overcomes the limitations of traditional static time-of-flight analysis by constructing trajectories from reaction time-of-flight data and embedding them into phase space with increased dimensionality. This effectively encodes the evolutionary trends, fluctuation patterns, and inertial characteristics of reaction behavior over time. By combining trend-fluctuation dual-domain construction, it achieves decoupled representation of long-term drift and short-term jitter in the reaction path, enhancing the model's ability to perceive potential attentional fluctuations. Through trajectory geometric quantization and change-point detection, it can accurately identify key state inflection points (such as the critical point from focus to fatigue), constructing high-order response representation features. This provides richer, more stable, and predictive event response data for attention modeling and fatigue level assessment, improving the system's analytical accuracy and discriminative efficiency for subtle behavioral changes.

[0060] Preferably, step S4 specifically includes:

[0061] Attention level index is calculated based on event response data to obtain event attention data;

[0062] Short-term trend aggregation is performed on event attention data to obtain attention trend feature data;

[0063] Fatigue is assessed based on event attention data and attention trend characteristics data to obtain personnel fatigue data.

[0064] This invention calculates an attention level index from event response data, enabling the construction of a time-series-meaning attention representation index based on multidimensional response features. This index accurately reflects an individual's focus and responsiveness during task execution. Combined with short-term trend aggregation, it effectively identifies subtle fluctuations and phased declines in attention levels. Finally, in the fatigue assessment stage, by fusing instantaneous attention index and short-term trend features, an interpretable fatigue assessment model is constructed. This not only improves the accuracy and real-time performance of fatigue identification but also enhances the system's ability to warn of sudden attention decline and potential fatigue risks, demonstrating strong practical value and scenario adaptability.

[0065] Preferably, this application also provides a fatigue detection system for personnel on long-range missions, used to perform the fatigue detection method for personnel on long-range missions as described above. The fatigue detection system for personnel on long-range missions includes:

[0066] The multi-source data acquisition module is used to acquire personnel image data and personnel voice data;

[0067] The multi-modal feature recognition and anchor embedding module is used for speech trigger event recognition according to personnel speech data, obtaining speech trigger event data; face dynamic feature extraction is performed according to personnel image data, obtaining face dynamic feature data; the face dynamic feature data is cross-modal anchor embedded to the speech trigger event data, obtaining multi-modal event data;

[0068] The reaction trajectory modeling and delay calculation module is used for reaction delay calculation on the multi-modal event data, obtaining event reaction data;

[0069] The attention evaluation and fatigue determination module is used for attention level processing according to the event reaction data, obtaining event attention data; fatigue determination is performed on the event attention data, obtaining personnel fatigue data.

[0070] The beneficial effects of the present application are that: by constructing an image and speech dual-modal acquisition path, high-fidelity acquisition of individual physiological behavior and task response data is realized. The dual-channel speech acquisition mechanism of the cabin audio device and the personal microphone in step S1, and the real-time image acquisition means in the cabin provide a complete, synchronous raw data basis for multi-modal analysis. In step S2, through cross-channel mutual verification recognition and double-layer keyword triggering, high-precision recognition of speech trigger events is realized, and combined with cross-site collaborative modeling in face dynamic feature extraction, speech events and face reactions are realized time sequence level fusion through cross-modal anchor embedding technology, generating multi-modal event data with clear structure and semantic coupling. In step S3, the multi-modal event data is input to construct a reaction trajectory, and high-order dynamic features that can describe the operation reaction agility and fluctuation are output by fusing phase space dimensionality, trend-fluctuation decoupling, trajectory geometric quantization and change point detection operations. In step S4, the attention index calculation and short-term trend aggregation are used to quantify the individual attention level fluctuation process, and the linkage recognition of short-term abnormalities and long-term trends is realized in fatigue determination, improving the accuracy, timeliness and adaptability of fatigue detection, which is suitable for continuous monitoring and intelligent scheduling in long-haul high-load task scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0071] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0072] Figure 1 A step flow chart of a long-haul task personnel operation fatigue detection method of an embodiment is shown;

[0073] Figure 2 A step flow chart of a multi-source data acquisition method of an embodiment is shown;

[0074] Figure 3 A step flow chart of a speech trigger event recognition method of an embodiment is shown;

[0075] Figure 4 A step flow chart of a reaction trajectory modeling and delay calculation method of an embodiment is shown.

[0076] Figure 5 A step flow chart of an attention evaluation and fatigue determination method of an embodiment is shown. DETAILED DESCRIPTION

[0077] The technical method of the present application will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0078] In addition, the drawings are only schematic illustrations of the present application, and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and thus repeated description thereof will be omitted. Some block diagrams shown in the drawings are functional entities, and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0079] It should be understood that although the terms "first", "second" and the like can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, a first element can be called a second element, and similarly a second element can be called a first element. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0080] Referring to Figures 1 to 5 The present application provides a long-haul task personnel operation fatigue detection method, comprising the following steps:

[0081] Step S1: obtaining personnel image data and personnel voice data;

[0082] Specifically, a monitoring camera is arranged in the cabin to collect facial image data of the personnel in real time at a sampling frequency of 15-30 frames per second, with an image resolution of no less than 720 pixels (720p). Meanwhile, the system is configured with two voice collection channels: one is a wide-area pickup device in the cabin as the first voice collection channel, with an audio sampling rate of 16 kHz, for capturing voice instructions and personnel communication statements in the cabin environment, and the collected audio is taken as the first personnel voice data; the other is a close-fitting wearable microphone as the second voice collection channel, with an audio sampling rate of 20 kHz, mainly for collecting clear voice of the speaker, and the collected audio is taken as the second personnel voice data. The system adds accurate time stamp markers to the collected image data and voice data.

[0083] Step S2: performing voice trigger event recognition according to the personnel voice data to obtain voice trigger event data; performing facial dynamic feature extraction according to the personnel image data to obtain facial dynamic feature data; and performing cross-modal anchor embedding of the facial dynamic feature data on the voice trigger event data to obtain multi-modal event data.

[0084] Specifically, the first personnel voice data and the second personnel voice data obtained as described above are subjected to time synchronization alignment processing. Short-time energy change characteristics and inter-frame voice similarity are taken as matching bases to perform sliding window matching between the two channels, so as to ensure that the voice data is aligned in the time dimension. In the aligned voice data, a speech recognition model is used for text recognition respectively. The speech recognition model is preferably a connection time sequence classification model based on a deep acoustic model or an end-to-end Transformer automatic speech recognition model based on an attention mechanism. For the double-channel recognition result, the system uses a mutual verification recognition strategy to enhance the recognition reliability: when the recognition texts of the two channels are consistent, or the text similarity exceeds 90%, the recognized voice text is directly confirmed as the effective trigger text; when there is a recognition difference between the two channels, the confidence (preferably the posterior probability score output by the speech recognition model) of each channel is weighted and fused to obtain the fused judgment text; if there is a fuzzy or low-confidence segment in the recognition result, a backup speech recognition model is enabled to re-execute inference and judgment to improve accuracy. Double-layer keyword trigger detection operations are performed on the confirmed text: the first layer detection performs preliminary screening based on acoustic characteristics, including indicators such as significant change in speech rate and sudden change in tone; the second layer detection is keyword-level text matching, and the matching content preferably includes instructive statements such as “report”, “in position”, “ready” and the like. If any of the above matching conditions is met, the corresponding voice trigger event data is output in combination with the corresponding time stamp.

[0085] In parallel, the system performs face dynamic feature extraction based on the personnel image data. This process includes the following steps: performing a face detection operation in the image frame, preferably using a multi-task cascaded convolutional neural network or a face recognition model such as RetinaFace, to locate the face region; in the detected face region, identify standard face key points, preferably sixty-eight key points, including eye, mouth, nose bridge, eyebrow, and other region feature points; extract eye-related features, including blink frequency, closed-eye duration, etc.; extract mouth-related features, including mouth opening angle, yawning duration, etc.; based on the time sequence action sequence of the eyes and mouth, perform dynamic coordination analysis to extract typical fatigue coordination behavior features such as "long-time closed eyes accompanied by mouth opening"; and encode the above key action sequence into a standardized face dynamic feature vector.

[0086] The system takes the timestamp of the aforementioned voice trigger event as an anchor point, and extracts the face dynamic feature sequence in the image frame sequence within about two seconds before and after the event, to complete the cross-modal time alignment processing. The face dynamic feature data is registered with the voice event time axis to generate voice and face image joint data with time consistency labels. The system encapsulates the above multi-modal joint data in a structured format to obtain multi-modal event data.

[0087] Step S3: Reaction delay calculation is performed on the multi-modal event data to obtain event reaction data;

[0088] Specifically, the trigger time point of each voice trigger event and the corresponding face response behavior starting time point are extracted from the encapsulated multi-modal event data. The voice trigger time point is the timestamp annotated when performing keyword trigger detection, indicating the time when the user's voice issues an instruction; the face response behavior starting time point is the detection time of the first behavior action (such as blinking, opening mouth, gaze conversion, etc.) in the face dynamic feature associated with the voice event. The reaction time difference between the voice event and the face behavior is calculated, denoted as the reaction delay value. The reaction delay value is defined as the face response starting time point minus the voice trigger time point; if the value is negative, it means that the face response is earlier than the voice trigger, which belongs to the pre-response behavior. For a plurality of continuous voice trigger events, the system constructs a reaction time trajectory sequence. Each trigger event corresponds to a reaction delay value, and all delay values are arranged in time sequence to form a reaction trajectory sequence; the sequence is used to describe the dynamic change pattern of the individual's reaction time within a period of time. For the reaction trajectory sequence, the system performs the following modeling processing steps: the reaction delay value at the current time and the values at the previous two time points are combined to form a three-dimensional embedding vector, which is used to describe the short-term historical context in which the current reaction state is located, for example, the current vector can be composed of "current delay value, previous delay value, previous two delay values". Before performing time series filtering, the system first uses the short-term historical context vector to judge the local stability of the current reaction state, and uses the result as the basis to adaptively set the filter parameters. Specifically, if the continuous reaction delay values in the context fluctuate less (such as the change rate is lower than the preset threshold), it is determined that the current state is relatively stable, and the filter uses a longer window length and a smoother threshold to retain the trend; if the fluctuation is larger (such as the standard deviation is higher), a shorter window and a stronger noise suppression parameter are used to more sensitively capture high-frequency changes. Based on the time series filtering method such as moving average or wavelet transform with the corresponding window parameters, the trend of the reaction trajectory is separated to distinguish the overall trend of slow change and high-frequency fluctuation. In the decoupled trajectory, geometric indicators are calculated, including local change rate (such as reaction speed or acceleration per unit time), coefficient of variation (i.e. the ratio of standard deviation to mean value), and curvature of the trajectory, which are used to quantify the stability and abnormality of the reaction behavior. Using a time series change point detection algorithm, key time points / critical points are identified in the reaction trajectory, which represent the initial appearance or sharp deterioration signal of the fatigue state. The system outputs the above analysis results as structured event reaction data, which includes but is not limited to the reaction time value of the current event; the geometric indicators of the current stage reaction trajectory; the overall change trend information (rising, falling or stable); and whether there is a variation point marker representing fatigue evolution.

[0089] Step S4: performing attention level processing according to the event reaction data to obtain event attention data; performing fatigue judgment on the event attention data to obtain personnel fatigue data.

[0090] Specifically, the attention index is formed by the following three types of features jointly weighted: (1) standardized reaction time: the value after maximum-minimum normalization processing of the reaction time difference; (2) fluctuation of reaction trajectory: the stability is measured by the coefficient of variation (representing the ratio of standard deviation to mean); (3) change curvature of reaction trajectory: used to describe the degree of local change in the trajectory. The system combines the above three indicators in a weighted form to obtain the attention index value, i.e. event attention data, and the calculation method is: Attention index = weight 1 (1 minus standardized reaction time) + weight 2 (1 minus coefficient of variation) + weight 3 (1 minus curvature value), wherein weight 1, weight 2 and weight 3 are the weighting coefficients of the three components, and all the indicators participating in the weighted calculation have been normalized to the [0, 1] interval, and preferably can be obtained by experience setting or training on historical sample data. The calculation of the attention index is based on a single event reaction, and each calculation is based on the complete time period of an event completion process, corresponding to a time length of 2-5 seconds, depending on the task type and operation process. The system can extract event attention data every fixed interval (such as every 10 seconds or every completed task module) in a continuous event sequence, form the attention index at multiple time points, and thus construct a dynamic trend trajectory within a time window, including the moving average of the attention index in the sliding window, the decline rate (i.e. the mean difference between consecutive windows), and the fluctuation frequency (i.e. the number of index fluctuations); if the attention index decreases each time in the last three time windows, and the decrease amplitude exceeds the preset threshold (such as 10%), the current state is marked as a trend attention decline. The system performs multi-level fatigue determination logic according to the current attention index value and its short-term trend characteristics: if the attention index is lower than 0.3 and shows a trend of decline, it is determined as moderate fatigue; if the attention index is lower than 0.2 and an abnormal change point (such as a sudden drop from stable state to unstable state) is identified in the reaction trajectory, it is determined as a severe fatigue state; the above determination conditions can be adjusted according to actual needs, or adaptive thresholds can be obtained by model dynamic learning. The system outputs personnel fatigue data. The fatigue data includes but is not limited to the current fatigue level, the attention change trend graph, the comparison analysis result with the historical state, the labeled change point position, etc.

[0091] Preferably, the personnel voice data includes first personnel voice data and second personnel voice data, and step S1 specifically comprises:

[0092] The personnel voice data is collected by the cabin audio device and the personal microphone, and the first personnel voice data and the second personnel voice data are obtained respectively;

[0093] Specifically, omnidirectional microphones are deployed above the operating area or driver's console, supporting sampling rates of at least 16 kHz and 16-bit audio depth to collect wide-area voice signals within the cabin, ensuring clear reproduction of human voice frequencies. These microphones employ a sound pressure threshold triggering mechanism, initiating acquisition when the detected sound pressure exceeds a preset threshold (e.g., 40 decibels), effectively filtering continuous background noise. The system integrates a voice activation detection module, which identifies the start and end boundaries of speech segments based on short-time energy and spectral subtraction algorithms, retaining only valid vocal segments. During acquisition, a timestamp is added to each segment of voice data, and the corresponding timestamp is recorded. The system acquires spatial direction information of the sound source (if the microphone array supports sound source localization or beamforming); each personnel wears a set of close-fitting microphones, preferably bone conduction microphones or high-fidelity chest microphones, with a sampling rate higher than 20 kHz, specifically designed for acquiring clear near-field speech data with strong anti-interference capabilities; the acquisition module is equipped with an automatic gain control mechanism to adapt to signal strength fluctuations caused by changes in wearing distance, ensuring stable speech acquisition; the system periodically evaluates the signal quality of the wearing device, and if data packet loss or a signal-to-noise ratio lower than a preset threshold (e.g., signal-to-noise ratio lower than 10 decibels) is detected, it automatically switches to a preset backup data source.

[0094] Personnel images are captured using in-cabin cameras to obtain personnel image data.

[0095] Specifically, a directional infrared high-definition camera is positioned in front of the operator or directly above the control panel. The camera has night vision and backlight suppression capabilities, with an image resolution of no less than 1280 x 720 pixels and a frame rate controlled between 15 and 30 frames per second. The system binds a timestamp to each frame of the image, with the accuracy preferably reaching the millisecond level. The camera's field of view covers the operator's head and upper body area, ensuring that facial, eye, and mouth features are completely within the visible range.

[0096] Preferably, the voice-triggered event recognition specifically includes:

[0097] Channel alignment is performed on the first person's voice data and the second person's voice data to obtain channel-aligned data;

[0098] Specifically, the voice data segments in the first channel (in-cabin wide-area pickup device) and the second channel (wearing microphone) are extracted respectively, and preliminary time alignment is performed according to their collection time stamps; in a set time sliding window (preferably two seconds), a dynamic time warping algorithm is performed on the audio frame sequences in the two channels to calculate the inter-frame similarity, which is used to align the speed difference or inconsistent start and end boundaries; if the misalignment or speed inconsistency of the two-way audio on the time axis is detected, the system performs frame-level registration through inter-frame interpolation, sliding window expansion and other methods, so that the two-way data is synchronized under the same time reference; after alignment, the structured channel alignment data is output, which includes the voice frame data sequence under the unified time axis, the original channel identifier (such as channel one or channel two) to which each frame belongs, and the alignment offset between channels.

[0099] Cross-channel mutual verification recognition is performed on the channel alignment data to obtain cross-channel fusion data.

[0100] Specifically, each voice segment after alignment is input into two independent automatic speech recognition models for transcription to obtain channel one text and channel two text, and the model is preferably a convolutional attention hybrid structure or an end-to-end large model (such as a speech recognition model based on a time convolution and attention fusion structure); the system performs consistency comparison processing on the two-channel transcription texts: if the texts are completely consistent, or the edit distance between them is less than a set threshold (such as less than two characters), it is marked as a high consistency paragraph; if there are differences between the texts, the different positions and contents of the key words or key phrases are compared; for the segments with conflicting recognition results, confidence weighted fusion is performed: the recognition posterior probability of each channel at this position is extracted as a confidence indicator, and the transcription result of the channel with higher confidence is preferentially retained; if the confidences of the two channels are similar and the difference is less than a set threshold (such as within five percentage points), a pre-set lightweight recognition model is called to re-identify, forming a re-discrimination mechanism; the system generates the fused text content and records the strategy label used for fusion, which includes but is not limited to "priority channel selection", "confidence weighted result retention" or "re-discrimination model correction".

[0101] Double-layer keyword triggering is performed on the cross-channel fusion data to obtain trigger detection data.

[0102] Specifically, the first layer: acoustic feature-based trigger detection, the system performs preliminary identification of the voice signal based on frame-level acoustic features, including but not limited to short-time energy, pitch variation amplitude, and speech rate variation rate; when continuous pitch rise, energy transition and other features are detected, and the pronunciation duration is shorter than the preset threshold (such as one second and five hundred milliseconds), it is determined that the segment may contain an instructional voice intent, and the corresponding time period is extracted as a trigger candidate interval. The second layer: keyword matching based on text content, keyword-level matching processing is performed in the fusion text, and the keywords come from a preset task instruction word table, preferably including "confirm", "report", "complete", "start", "prepare", "receive" and the like; the system locates the clause containing the keyword in the text, extracts the occurrence timestamp and the position of the keyword in the sentence, and forms a text trigger candidate set. Compare the first layer acoustic trigger candidate time period with the second layer text keyword occurrence time period; if there is more than 50% overlapping interval on the time axis, it is determined as an "effective trigger" event; otherwise, it is marked as a "suspected trigger" for subsequent review or secondary confirmation processing. The system outputs structured trigger detection data, including but not limited to matched keywords, recognition confidence (the recognition confidence is based on the time overlap of acoustic trigger and text trigger and the text similarity of keyword matching, combined with the consistency of cross-channel recognition results to form a weighted calculation, and the input parameters need to be standardized before calculation during the calculation process), trigger type label (such as valid or suspicious), and corresponding time information.

[0103] Generate voice trigger event data according to the trigger detection data.

[0104] Specifically, the voice event recognized as "effective trigger" is structured and converted to generate the corresponding semantic label; each voice trigger event data contains the following field information: event type: according to the keyword content to analyze the event semantic category, if the keyword is "confirm", it is marked as "operation confirmation class event", if it is "complete", it is marked as "task completion class event", etc.; trigger time: extract the timestamp information matched with the event, preferably from the acoustic detection result or the synchronous time record in the speech recognition system; recognition channel source: mark the main channel used by the speech recognition result, such as channel one (in-cabin pickup) or channel two (wearing microphone); confidence score: according to the cross-channel fusion recognition and double-layer trigger verification result, calculate the recognition confidence of the event, which is used to evaluate the result reliability (i.e. the recognition confidence is obtained by comparing the consistency score of the speech recognition text of each channel, and combining the double-confirmation result of the trigger event in acoustic detection and semantic recognition to obtain the recognition confidence of the event); voice text content: keep the complete sentence content corresponding to the trigger event in the original recognition text as the semantic basis of the event.

[0105] Preferably, the cross-channel mutual verification identification specifically includes:

[0106] Perform single-channel speech recognition on the channel-aligned data to obtain speech recognition data;

[0107] Specifically, the aligned first-channel speech data (i.e., the cabin environment sound pickup signal) and second-channel speech data (i.e., the wearable microphone signal) are respectively input into the automatic speech recognition model for recognition. The speech recognition model can preferably adopt a convolutional-attention hybrid structure model based on the connection-time classification (CTC) mechanism, or an end-to-end recognition model based on the Transformer architecture. The recognition operation is performed independently for each channel of speech data, and the system outputs: the transcribed complete speech text; the recognition confidence score corresponding to each word or sub-word unit, preferably using posterior probability as the evaluation index; and the time alignment information corresponding to each text unit.

[0108] Consistency comparison is performed on the speech recognition data to obtain consistency comparison data;

[0109] Specifically, the system extracts the speech recognition text corresponding to the first and second channels and performs comparative analysis on them. It constructs the shortest operation path between the two text segments, converting one segment into the other; the minimum number of editing operations required is the edit distance value. The system calculates the similarity score between the recognized texts based on the edit distance and text length. The calculation method is: divide the "edit distance" by the larger of the two text lengths, and then subtract that ratio from 1 to obtain the similarity score. If the similarity score is greater than 0.9, it is considered highly consistent; if the similarity score is less than 0.7, it is marked as a low-consistency segment, indicating potential recognition bias or semantic conflict, requiring subsequent manual or model intervention. The system outputs consistency comparison data, including the similarity scores of the two text segments and the index positions of differing characters or words.

[0110] Based on the consistency comparison data, the difference location is obtained by locating the differences.

[0111] Specifically, inconsistent words or phrases in two recognized text sequences are compared item by item to determine the start and end positions of the differing segments; the corresponding timestamp information of the differing segments in the original speech is extracted as the basis for time localization; at the same time, the recognition confidence score of the segment in both channels is obtained, preferably using posterior probability as a metric to judge the recognition reliability of each channel for the segment; the difference localization data includes the start and end position indices of the differing segments; the corresponding time range of the differing segments; and the recognition confidence score of each channel for the segment.

[0112] Based on the differential positioning data, a confidence-weighted discrimination is performed to obtain weighted discrimination data;

[0113] Specifically, in the located difference segment, the system weights the recognition confidence of the difference word or phrase according to each channel; for each candidate word, the recognition confidence scores provided by the two channels are counted respectively, and a weighted calculation is performed in combination with the channel confidence preference weight, which can be set as a fixed value (such as 1:1) or adaptively adjusted according to the historical recognition stability; the system compares the weighted results, retains the recognition text with higher confidence weighted score on one side as the final determination result of the current segment; if the confidence difference of the two channels in the segment is less than a preset threshold (preferably 0.05), the segment is marked as “uncertain”, and a standby recognition model is triggered to perform a re-determination operation.

[0114] According to the weighted discrimination data, re-discrimination is performed to obtain re-discrimination data.

[0115] Specifically, for the difference segment marked as “uncertain”, the system re-extracts the corresponding original audio data; the audio segment is input into an independently configured lightweight speech recognition model for re-recognition, and the model is preferably an RNN-Transducer model based on a recurrent neural network structure or an identification algorithm based on acoustic feature template matching; optionally, the system enables a multi-lingual or homonym confusion word detection module in the re-discrimination process to identify common homonym words or confusing words (such as “receive” and “know”) in Chinese speech, and to modify the recognition result in combination with a preset confusion word replacement set; if the re-discrimination output result is consistent with the text in the original recognition channel, it is confirmed that the original recognition is valid and is retained; if there is a difference, the output result of the re-discrimination model is used to replace the original recognition content.

[0116] The re-discrimination data is fused across channels to obtain cross-channel fusion data.

[0117] Specifically, for the text segment determined to be highly consistent in the consistency comparison stage, the system directly includes the segment into the fusion result; for the segment with a difference, the system optimizes and retains according to the confidence weighted discrimination result or the output result of the re-discrimination model; after completing the text content fusion, the system regenerates the time alignment information to ensure the consistency of the fusion text and the original speech data in time sequence: if the fusion result is derived from the second channel (such as a wearable microphone) recognition result, the timestamp information of the channel is directly inherited; if the fusion result is derived from the output of the re-discrimination model, the corresponding time alignment label is regenerated according to the audio processing frame rate and the recognition output step of the model; the system outputs a structured fusion speech recognition result, which includes the complete fused speech recognition text; the source channel or recognition model corresponding to each text segment; and the fusion strategy label used, such as “channel priority retention”, “re-discrimination correction”, “consistent segment direct merging”, etc.

[0118] Preferably, the double-layer keyword trigger specifically refers to:

[0119] Acoustic layer candidate detection is performed on the cross-channel fusion data to obtain acoustic layer candidate data;

[0120] Specifically, a raw speech waveform segment of the last three to five seconds is extracted from the fusion speech recognition result, and frame-level audio analysis is performed on the segment. During the audio analysis process, the system extracts low-level acoustic feature parameters, preferably including but not limited to short-time energy features, which are used to reflect the energy change of audio within a local time window; pitch contour features, which are used to describe the continuous change pattern of pitch over time; mel-frequency cepstral coefficients (MFCC), which are used to extract the spectral features of speech; and speech rate indicators, i.e., the number of recognized words per unit time, which are used to assess the pace of speaking. Based on the above features, the system builds an acoustic event determination model to screen potential trigger segments through combination rules, which preferably include a rapid rise in audio energy within a short time and a duration less than a preset threshold (such as two seconds); a clear upward trend in pitch and a short-time mutation passage. If a speech segment simultaneously satisfies multiple acoustic feature combination conditions, the segment is marked as an “acoustic candidate trigger segment”; the system outputs structured acoustic layer candidate data, which includes the start and end times of the candidate trigger segment, and the corresponding acoustic feature labels.

[0121] Text layer keyword detection is performed on the cross-channel fusion data to obtain text layer keyword data;

[0122] Specifically, complete recognized text sentences are extracted from the fusion speech recognition result, and timestamp information corresponding to each word is synchronously obtained, preferably based on a word-level time alignment mechanism. The system has a built-in task-related keyword list for matching key expressions that trigger semantic events, which preferably includes “confirm”, “complete”, “receive”, “start”, “report”, “execute”, “prepare”, etc. The system performs word segmentation processing on the fusion text and performs keyword matching based on preset rules, supports using regular expression matching mode and semantic variant recognition mechanism to realize the recognition ability of extended expressions such as “has confirmed” and “ready”. For each successfully matched keyword, the system records the following information fields: the matched keyword content; the starting position of the keyword in the original sentence, which is positioned using character-level indexing; the timestamp corresponding to the keyword, which is inversely calculated based on the word-level alignment information; the context paragraph information of the sentence, which is used to support subsequent context semantic analysis and fusion judgment. The system outputs structured text layer keyword data, which includes keyword text, occurrence position, time information, and context sentence content, etc.

[0123] Cross-layer consistency verification is performed on the acoustic layer candidate data and the text layer keyword data to obtain cross-layer consistency data;

[0124] Specifically, the system compares the acoustic layer candidate segment with the time interval of the sentence segment where the text layer keyword is located, calculates the overlap ratio of the two on the time axis, and determines that the segment is "cross-layer consistent" if the overlap length of the acoustic layer candidate segment and the time segment where the text keyword is located accounts for more than or equal to 50% of the total length of the joint time segment, indicating that the acoustic features and semantic triggers are basically synchronous. If the offset of the two on the time axis exceeds a preset threshold (such as two seconds), it is marked as "cross-layer inconsistent", indicating that there is an asynchronous recognition or false trigger. Optionally, the system performs a speech speed balancing verification mechanism, for example, by analyzing the time interval of the words before and after the text keyword to determine whether the speech speed conforms to the normal speech flow distribution, and if the inter-word pause is too long, it is considered abnormal. The system outputs structured cross-layer consistency data, which includes fields such as time overlap ratio, offset, consistency flag, and optional speech speed balancing verification results.

[0125] The cross-layer consistency data is subjected to trigger confidence evaluation to obtain trigger detection data.

[0126] Specifically, the system performs multi-factor scoring on the candidate event based on acoustic layer confidence, text layer keyword confidence, and cross-layer time alignment degree. The acoustic layer confidence can be extracted from the mutation features of the audio signal, preferably including indicators such as pitch change amplitude and energy surge ratio. The text layer confidence can be quantified by the posterior probability output by the automatic speech recognition model to quantify the confidence of the recognized keyword. The cross-layer time alignment score is used to evaluate whether the acoustic features and keyword semantics are highly synchronized on the time axis. The system constructs a joint confidence scoring function, normalizes and standardizes the three types of scores according to the set weight, and combines them to generate a trigger confidence score. The weight parameter can be configured, and the default setting is preferably: the acoustic layer confidence weight is 0.4, the text layer confidence weight is 0.4, and the time alignment score weight is 0.2. If the final trigger confidence score is higher than the set high threshold, it is marked as "valid trigger event". If the score is in the middle threshold interval, it is marked as "suspected trigger event". The system outputs structured trigger detection data, which includes joint trigger confidence score, confidence score of each dimension, flag information whether it is determined as a valid trigger, corresponding voice time range and keyword content.

[0127] Preferably, the face dynamic feature extraction specifically includes:

[0128] Face detection is performed on the personnel image data to obtain face detection data.

[0129] Specifically, the system performs a face detection operation on the image frame sequence collected by the camera, preferably using a face detection model constructed with a lightweight convolutional neural network structure, such as RetinaFace, MTCNN or BlazeFace, to meet the real-time detection requirement; the system sets a face detection frequency of at least five frames per second in the image stream and enables a continuous frame tracking mode to track the detected face target across frames; for each frame detection result, the system outputs the bounding box information of the corresponding face region, including the top-left corner coordinate position and width-height data, to describe the spatial distribution of the face in the image; at the same time, the confidence score of face detection is output, preferably only retaining the detection result with a confidence score not lower than 0.85; the system also records the timestamp information of each frame detection result.

[0130] Perform facial key point recognition on the face detection data to obtain facial key point data.

[0131] Specifically, within the detected face region, the system calls a face key point detection model to perform fine recognition on the facial key region, preferably using a face annotation model that supports extraction of 68 or 106 key point coordinates; the extracted key points include but are not limited to left and right eye corners, eyebrow apexes, nose tip, upper and lower lip contours, cheek edges, etc., and the coordinate information is output in the form of pixel point positions in the image space; the system performs time dimension smoothing processing on the position of each key point, preferably using an exponential moving average algorithm to fuse the key point coordinates of the current frame and the previous frame to reduce detection jitter and noise interference; the system outputs structured facial key point data, including the spatial coordinate information of each key point in each frame and the time-smoothed position sequence.

[0132] Perform eye feature extraction and mouth feature extraction according to the facial key point data to obtain eye feature data and mouth feature data, respectively.

[0133] Specifically, the system extracts eye features and mouth features according to the facial key point data respectively, and outputs eye feature data and mouth feature data. Eye feature extraction: the system selects six key points of the eye region in the recognized facial key points, which correspond to the eye corner and upper and lower eyelid positions respectively; based on the six key point coordinates, the eye opening ratio is calculated, which is used to measure the ratio of the vertical opening degree to the horizontal width of the eye; if the calculated eye opening ratio is lower than a set threshold (such as 0.2) and the duration of this state exceeds 300 milliseconds, it is determined as a closed-eye behavior; the eye feature data output by the system includes the EAR value of the current frame; the determination result of whether there is a blinking event; the duration statistics data of the closed-eye state, preferably using a sliding time window for cumulative judgment. Mouth feature extraction: the system selects the key point coordinates of the upper and lower edges of the lips and the corner of the mouth, calculates the mouth opening ratio, which is used to measure the proportion of the vertical opening amplitude to the horizontal width of the mouth; when the mouth opening ratio is higher than a set threshold (such as 0.6) and the duration of this state exceeds 800 milliseconds, it is determined as a yawn behavior; the mouth feature data output by the system includes the MAR value of the current frame; the detection flag of whether there is a yawn behavior; the duration information of the mouth opening state.

[0134] According to the eye feature data and the mouth feature data, cross-site correlation processing is performed to obtain cross-site correlation data.

[0135] Specifically, the system constructs a time sequence matrix of cross-site actions based on consecutive image frames, each frame constitutes a state vector, which includes the following four contents: the EAR of the current frame; the MAR of the current frame; the Boolean flag of whether the current frame has a blinking behavior; the Boolean flag of whether the current frame detects a yawn behavior. In the above time sequence, the system performs synchronous analysis operation of cross-site behavior, and identifies potential collaborative fatigue action: if the start and end time difference of closed-eye behavior and mouth opening behavior is less than or equal to 0.5 seconds within a certain time period, and the overlap ratio of the two on the time axis exceeds 60%, it is determined that this paragraph is an effective collaborative fatigue action, which is regarded as a collaborative fatigue event; the system counts the following correlation indexes: the frequency of collaborative fatigue events, i.e. the number of collaborative events per unit time; the average duration of each collaborative event, which is used to evaluate the stability of the fatigue state; the time distribution density of the collaborative event within the whole monitoring time, which is used to judge whether there is a stage concentration trend. The system outputs structured cross-site correlation data, which includes the start and end time, duration, frequency, and time distribution characteristics of the collaborative event.

[0136] The cross-site correlation data is mapped to generate collaborative mode, and facial dynamic feature data is obtained.

[0137] Specifically, the system constructs a behavior pattern encoder for mapping the detected sequence of collaborative fatigue events into corresponding dynamic state labels; different types of facial behavior combinations are assigned different behavior encodings, preferably using the following mapping strategy: a single blink behavior is encoded as state label "E1"; a collaborative event of closing eyes and opening mouth occurring within the same time period is encoded as "E3", representing a general fatigue collaborative state; if more than three mouth opening behaviors with significantly higher mouth opening than normal occur consecutively within a short period of time, accompanied by at least one closing eyes behavior, it is determined as a high-risk fatigue state, encoded as "E5". The system performs frequency statistical analysis based on the sequence of facial behavior states within the last 5 seconds, extracts the most frequently occurring behavior encoding within the time window as the current facial fatigue state; the system outputs structured facial dynamic feature data, which includes time window, main behavior encoding label, behavior frequency statistical result, etc.

[0138] Preferably, the cross-modal anchor embedding is specifically:

[0139] Voice event anchor point extraction is performed on the voice trigger event data to obtain voice event anchor point data;

[0140] Specifically, the structured voice event data output by the voice trigger event recognition module is read, which includes keyword content, recognition confidence, complete sentence text, and timestamp information corresponding to the trigger event; the system takes the time point when the keyword first appears in the sentence as the anchor point center time of the voice event, preferably, this time point can be deduced from the word-level alignment result of voice recognition; for example, if the keyword "confirm" first appears at 14:23:10:580, the anchor point center time is set to this time point; On this basis, the system sets the anchor point time window parameter for determining the range of action of the anchor point on the time axis, preferably set to 1.5 seconds before and after the anchor point, i.e. the total time window is 3 seconds; the time window can be dynamically adjusted according to the average response delay after the voice trigger in the historical samples; the system generates structured voice event anchor point data, which includes anchor point center time; time window range; associated keywords and sentence text.

[0141] According to the voice event anchor point data, the facial feature segment is intercepted to obtain the facial feature segment data;

[0142] Specifically, the system obtains the generated facial feature time series data from the facial dynamic feature extraction module, which is recorded in image frame order, and the content includes the eye aperture ratio (EAR) of each frame, the mouth aperture ratio (MAR), the fatigue event code, etc.; for each speech event anchor point, the system reads its corresponding time window start and end points, i.e., the anchor point start time and end time; the system retrieves and extracts the continuous frame-level feature data in the time period in the facial feature time series according to the above time interval, for example, when the facial image frame rate is 25 frames per second and the anchor point time window length is 3 seconds, the system will intercept a continuous feature segment of 75 frames; during the interception process, the system retains the timestamp information corresponding to each frame and all facial feature values contained in the frame; the system outputs structured facial feature segment data, which is arranged in time order, and the content includes timestamp, EAR / MAR value, cooperative state code, etc. Field, which can be one-to-one corresponding to the speech trigger event.

[0143] According to the speech event anchor point data and the facial feature segment data, the time sequence alignment data is obtained.

[0144] Specifically, the system aligns the timestamp of each frame of facial feature data in the facial feature segment with the corresponding speech event anchor point time. Take the anchor point time of the speech event as the center reference time, and convert each frame timestamp into a relative time value relative to the anchor point, for example, the image acquisition time of a certain frame minus the anchor point time, which is the relative time of the frame. The system adds a "relative time" field in each frame of facial feature data. Optionally, if there is a time resolution difference between the facial image frame rate and the speech event related data (such as ASR output), the system can perform time interpolation or downsampling processing on the facial feature data to achieve alignment under the unified time step. The system outputs structured time sequence alignment data, which contains facial feature values, relative time information and corresponding anchor point number identification of each frame.

[0145] According to the time sequence alignment data, the anchor point embedding is performed to obtain the multi-modal event data.

[0146] Specifically, the system constructs an integrated multi-modal event representation unit of "anchor point-behavior" based on each unit of voice event data combined with the corresponding time-aligned facial feature segments; in the embedding process, the system fuses the voice and facial modal features in one of the following two ways: 1. The structured attribute information of the voice event (including keyword content, recognition confidence, trigger time, etc.) is spliced with the corresponding frame-level facial feature data (such as eye opening ratio EAR, mouth opening ratio MAR, fatigue state code, etc.) within the anchor point window to form a unified multi-field structure for describing the language and visual behavior state under the same time anchor point; 2. Statistical feature extraction is performed on the facial feature sequence within the anchor point time window to generate an abstract vector for modeling, preferably including but not limited to the maximum mouth opening ratio; the cumulative number or total duration of eye closure behavior; the frequency of occurrence of fatigue-related state labels (such as collaborative coding) within the window. The system outputs structured multi-modal event data, which is in units of "anchor points" and fuses voice information and facial behavior features.

[0147] Preferably, step S3 is specifically

[0148] Trigger anchor point time is obtained for the multi-modal event data to obtain trigger anchor point time data.

[0149] Specifically, the system extracts the timestamp field of the voice trigger event from the multi-modal event data that has completed embedding processing, and this field is the trigger time marker; if the voice event is composed of continuous sentences, the system selects an appropriate time between the first occurrence time of the keyword and the end time of the sentence as the anchor point time according to the requirements of the application scenario: when the system needs to detect "reaction behavior after issuing an instruction", the time when the keyword first appears in the sentence is preferably used as the anchor point time; when the system needs to analyze "confirmation feedback behavior", the time at the end of the sentence is preferably used as the anchor point time; the system uniformly generates trigger anchor point time data in a standard format for each multi-modal event, and the data includes anchor point time value; time type identifier (such as "keyword start time" or "sentence end time"); corresponding voice event identifier code or unique index.

[0150] Response behavior time is captured for the trigger anchor point time data to obtain response behavior time data.

[0151] Specifically, the system takes the trigger time in each trigger anchor point time data as the starting point, searches backward for a preset time window, preferably 2 seconds, to capture possible facial response behaviors; the type of response behavior is a configurable parameter, including but not limited to eye behavior: detecting whether there is a first eye closing action or a blinking event; if the duration of the eye closing behavior is detected to be more than 300 milliseconds, it is considered as an effective response behavior; mouth behavior: detecting whether the mouth opening degree exceeds the set threshold, such as mouth opening ratio (MAR) greater than 0.5; or detecting whether there is a "yawn" state label, marked as fatigue behavior in previous behavior coding; if any of the above valid behaviors is detected within the set time window, the system records the occurrence time of the first valid behavior as the response time, and labels the response type (such as "blinking" or "yawn"); if no response behavior meeting the conditions is detected within the time window, the system marks the event as "no response" state; the system outputs structured response behavior time data, including response behavior type; response occurrence time; response hit; associated trigger anchor point identification information.

[0152] Reaction time difference data is calculated by reacting to the response behavior time data;

[0153] Specifically, for each voice trigger event, the system extracts the trigger time and facial response time corresponding thereto, calculates the time difference between the two as the reaction delay time of the event; the reaction time difference is calculated by subtracting the trigger time from the response time to obtain the delay value, with the unit of milliseconds; the system performs boundary judgment on the calculation result to mark abnormal response situations: if the reaction time difference is less than 0, the response behavior has occurred before the voice trigger, which is marked as "pre-response" state, reflecting inertial action or system noise misjudgment; if the reaction time difference is greater than 3 seconds, the response is too late, which is marked as "super-late response", indicating insufficient attention or fatigue state; at the same time, the system records in each reaction time difference data: whether an effective response is detected; the type of response behavior (such as blinking, yawning, etc.); the actual reaction time difference value, with the unit of milliseconds.

[0154] Reaction feature extraction is performed according to the reaction time difference data to obtain event reaction data;

[0155] Specifically, the system constructs a reaction trajectory time series, and arranges reaction time differences of multiple voice trigger events in time sequence as a set of continuous numerical values to form a reaction trajectory sequence; the system performs multi-dimensional feature analysis on the sequence, specifically including: 1. The sliding window method is used to calculate the mean value variation trend of the reaction time difference in different time periods to represent the incremental difference value between the subsequent window and the preceding window; according to the set sliding step, the local slope of the reaction time difference sequence is calculated to determine whether there is a sustained delay rise in the current stage; if the slope value is greater than 200 milliseconds per event on average, it can be considered as a trend of attention level decline, indicating a potential fatigue state. 2. The standard deviation of the entire reaction trajectory is calculated; the coefficient of variation (i.e. the ratio of the standard deviation to the average value) is calculated to standardize the evaluation of the stability of the reaction; at the same time, the maximum jump amplitude of the reaction time difference between adjacent events is recorded to identify the dramatic response fluctuations. 3. Behavior mutation detection is performed to analyze possible mutation points in the reaction trajectory, such as using cumulative sum control chart (CUSUM), Mahalanobis distance analysis, or topological method based on low-order persistent homology to identify local abnormal paragraphs, judge whether there is a significant change in behavior pattern, and obtain a qualitative evaluation label. Based on the above multi-dimensional features, the system generates structured event reaction data, which includes but is not limited to reaction trend slope; reaction trajectory volatility indicators (standard deviation, coefficient of variation, maximum jump); whether there is a behavior mutation point and its location; qualitative evaluation label of overall attention stability.

[0156] The reaction feature extraction is specifically:

[0157] According to the reaction time difference data, the reaction trajectory data is obtained by constructing the reaction trajectory;

[0158] Specifically, the system extracts the reaction time difference data corresponding to the multiple processed multi-modal events from the multiple processed multi-modal events, that is, the time interval between the voice trigger time and the response behavior occurrence time in each event, in milliseconds or seconds; the above multiple reaction time difference values are arranged in time sequence to form a one-dimensional numerical sequence, which is used to represent the reaction behavior change trajectory evolving over time; the system can set a sliding window strategy to limit the length of the reaction trajectory, for example, only the reaction time difference data of the last 10 events are retained; the system outputs structured one-dimensional reaction trajectory data.

[0159] The reaction trajectory data is embedded in the phase space to obtain reaction dimensionality data;

[0160] Specifically, the system uses the delay coordinate reconstruction method to map the original one-dimensional reaction trajectory time series to a multi-dimensional state space; during the dimensionality increasing process, the system automatically selects appropriate embedding dimension and time delay parameters according to the autocorrelation function or the false nearest neighbor algorithm, for example, the embedding dimension can be set to 3 and the time delay step can be set to 1, or it can be optimized and adjusted according to different application scenarios; for each time point, the system constructs a multi-dimensional state vector through delay coordinates: the vector is composed of the current reaction time difference, the reaction time difference after one-step delay, the reaction time difference after two-step delay, and so on, arranged in time sequence, until all components of the preset embedding dimension are included; the original one-dimensional sequence is unfolded in the multi-dimensional state space, so that the adjacent state vectors represent the reaction dynamic evolution state of the user in the time period; the system outputs the reaction dimensionality increased data, which is a set of multi-dimensional state vectors arranged in time sequence.

[0161] According to the reaction dimensionality increased data, a trend fluctuation double-domain construction is performed to obtain double-domain construction data;

[0162] Specifically, the trend domain construction: the system performs moving average processing or smoothing fitting method based on local weighted regression (such as Loess regression) on each dimension of the reaction trajectory data after dimensionality increasing, to extract the slow trend curve; the system calculates the slope change of adjacent time periods in the time series, and calculates the trend change amplitude in unit time in a fixed step time sliding window manner, which is used to quantify the strength of the trend rise or fall; for example, the first-order difference method can be used to compare the trajectory value at the t time with the trajectory values at the previous k time points, to obtain the trend change rate; the trend domain data is output, which includes the trend value and the local change slope at each time point, for describing the long-term fatigue trend or attention drop phenomenon. Fluctuation domain construction: the system analyzes the residual sequence between the trajectory after dimensionality increasing and its corresponding trend curve (such as slow trend curve), to extract high-frequency fluctuation information; in the residual sequence, the system calculates the following indexes: coefficient of variation: the ratio of standard deviation to mean value, used to measure the relative strength of fluctuation; maximum jump amplitude: the maximum amplitude of trajectory value change between adjacent time points, used to capture the sudden behavior reaction; local oscillation rate: the frequency of directional change in unit time, used to represent high-frequency behavior instability; the fluctuation domain data is output, including the fluctuation intensity index, the maximum jump value, etc. The above trend domain data and fluctuation domain data are labeled and combined into double-domain construction data.

[0163] According to the double-domain construction data, trajectory geometry quantization is performed to obtain trajectory geometry data;

[0164] Specifically, the system quantifies the geometric characteristics of the motion path of the user in the reaction state space based on the dimensionally upgraded trajectory data, and obtains trajectory geometry data. Specifically, for any time point in the dimensionally upgraded state trajectory, the system estimates the local bending degree of the trajectory using a three-point approximation method. With the time point and its adjacent two state vectors as references, the difference between the lengths of the front and rear sections of the path is calculated, and then normalized with the overall span to obtain the local curvature index of the position. The index is used to represent the smoothness and mutation degree of the behavior state change. The total path length refers to the overall movement distance of the trajectory in the state space, which is obtained by calculating the Euclidean distance between adjacent state vectors point by point and summing them up, and is used to measure the intensity of the overall reaction change. The trajectory variation rate is used to reflect the frequency of the change in the curvature of the trajectory. The calculation method is to measure the curvature change at each time point, and sum the change sequence. The greater the value, the more intense the switching of the behavior state. The system also calculates the retreat ratio index of the trajectory. The retreat ratio index is the ratio of the maximum downward segment amplitude to the adjacent maximum upward segment amplitude in the trajectory, reflecting the asymmetry of the rapid decay and recovery of attention in a short time. The index can reveal whether the user has a rapid recovery behavior after "sudden loss of attention" or exhibits a persistent low response state. If the retreat ratio value is high (greater than a preset empirical parameter or a preset parameter), it indicates that the user state change has obvious nonlinear fluctuation characteristics. The above indexes are labeled on the dual-domain construction data to obtain trajectory geometry data.

[0165] The trajectory geometry data is subjected to change point detection to obtain event reaction data.

[0166] Specifically, the system performs multi-dimensional fusion type change point detection operation based on double domain construction data (including trend domain and fluctuation domain indicators) and trajectory geometry data (including curvature, trajectory length, variability rate, drawdown ratio, etc.), identifies key behavior abnormal moment in the user reaction process, and generates event reaction data. The trend domain data includes the trend value and local trend slope at each time; the fluctuation domain data includes the fluctuation intensity indicator (such as coefficient of variation CV, jump amplitude, oscillation frequency) at each time; the trajectory geometry data includes the local curvature, cumulative path length, curvature change rate, drawdown ratio, etc. at each time. The system identifies the key time point of behavior turning or attention change in the trajectory through the following multi-dimensional collaborative strategy: when the trend slope mutates in a plurality of consecutive time steps (such as from rising to falling in a short time, and the change amplitude exceeds the set threshold), it is judged as a potential change point. If the coefficient of variation in the fluctuation domain significantly increases in a unit of time, and accompanied by a significant increase in the maximum jump value, it indicates that the behavior state becomes unstable, which is a change point candidate. In the trajectory geometry data, if the curvature value of a plurality of consecutive time points suddenly increases, or the curvature change rate exceeds the set threshold (such as >0.1), it can be considered as a signal of local trajectory bending and behavior switching, and marked as a "reaction abnormal point". If it is observed that there is an obvious "rise-sudden drop-rise" fluctuation mode before and after a certain time point, and the drawdown ratio exceeds the set proportion (such as >0.7), it is considered that the user has experienced a large fluctuation of attention or response state. The system uses a combined strategy to screen and confirm the candidate change points. Only when at least two of the above indicators meet the set conditions in the same time period, the system marks the time as a "change point". The output event reaction data includes the timestamp of each change point, trigger indicator source (such as trend mutation and high-frequency fluctuation), change point type (sudden increase, sudden decrease, oscillation), etc.

[0167] Preferably, step S4 is specifically:

[0168] According to the event reaction data, the attention level index is calculated to obtain event attention data;

[0169] Specifically, the system first extracts the reaction feature data corresponding to each event, mainly including the following three key indicators: reaction time difference: the delay between the reaction time and the trigger time of the event (unit: milliseconds), which is standardized by the maximum time difference; trajectory stability indicator: the coefficient of variation of the trajectory reaction value, or the normalized standard deviation value, which is used to measure the consistency and fluctuation amplitude of the reaction; response behavior type: refers to the specific facial response type detected by the system, such as "wink", "open mouth" or "no response", etc. According to the above feature data, the system calculates the attention level index of the event, and the calculation formula is as follows: the reaction time difference is standardized to a proportion between 0 and 1; the stability indicator is expressed as a normalized score; different behavior types are assigned behavior scores, for example, if it is a "wink" behavior, the behavior score is 0.5; if it is a "mouth open" behavior, the score is 0.3; if no response is detected (i.e. "no response"), the score is 0. The attention index can be expressed as: attention index = a x (1 - standardized reaction time difference) + β x stability score + γ x behavior score. Wherein, α, β, γ are the weighted coefficients of each indicator, and the sum of the three is 1. For example, the system can set α = 0.6, β = 0.3, γ = 0.1, to emphasize the main role of reaction delay in attention evaluation. The obtained attention level index value range is [0, 1], i.e. event attention data, where the higher the value, the more concentrated the attention state, and the lower the value, the more likely there is a lack of concentration or fatigue response in the event.

[0170] According to the event attention data, short-term trend aggregation is performed to obtain attention trend feature data;

[0171] Specifically, the event attention data is subjected to short-term trend aggregation analysis to identify the attention trend state of the current stage of the user, and generate attention trend feature data. Specifically, the system extracts the corresponding attention level index values from the last n events to form time-ordered attention sequence data. The window length n can be set to a fixed value (such as 10), or can be dynamically adjusted according to the task scenario. On the basis of the constructed sliding window, the system performs the following three trend feature modeling operations: calculating the average value of all attention indices in the sliding window to reflect the overall attention level of the stage; estimating the attention change slope in the short term by the difference between the attention indices of the current time and the previous time, to judge whether the attention is rising, falling or remaining stable. Calculate the standard deviation of the attention index values in the sliding window as an attention volatility indicator; if the standard deviation exceeds a preset threshold (for example, 0.15), it is determined that the attention fluctuation in the current stage is relatively severe, indicating that there is an unstable or dispersed attention situation. The system detects whether there are multiple events with low attention indices in the sliding window; if the attention indices of three consecutive events are lower than a preset lower limit (such as 0.4), the trend is marked as "attention fatigue danger trend", prompting the user or the system to take active intervention. The system structures the above trend results as attention trend feature data.

[0172] According to the event attention data and the attention trend feature data, fatigue is determined to obtain personnel fatigue data.

[0173] Specifically, based on the event attention data and the attention trend feature data, the fatigue state of the personnel is determined, and a fatigue state label or grade is outputted to form personnel fatigue data. The system predefines a plurality of determination rules for identifying fatigue risk, and the rules can be realized in the form of rule tree, soft logic or configurable parameters, and specifically include: if the current attention index is lower than a preset threshold (such as 0.4), and the attention trend is in a downward state (i.e. the attention index continuously decreases in a short period), it is determined as "suspected fatigue state"; if the attention index is lower than 0.5 for three times continuously in a sliding window, it is also regarded as having fatigue risk; if the attention fluctuation rate of the current stage is high (such as the standard deviation is greater than 0.2), and the average value of the attention index is low (lower than 0.55), it indicates that there is attention instability and fatigue sign. Any one of the above conditions satisfies can trigger fatigue state alarm or prompt. In order to finely judge the personnel state, the system can divide different fatigue grades according to the attention index level and trend feature, including: attention full state: the current attention index is significantly higher than the normal value (such as higher than 0.75), and shows an upward trend, indicating that the personnel's mental state is good; normal state: the current attention index is in the middle range (such as between 0.5 and 0.75), and the state is stable; fatigue critical state: the attention index is close to the lower limit of the fatigue threshold (such as between 0.4 and 0.5), and shows a downward trend in a short period, prompting fatigue precursor; fatigue confirmation state: the attention index continuously decreases below the minimum warning line (such as lower than 0.4), and has appeared a downward trend for three times continuously, which clearly indicates that the personnel has entered the fatigue state. The system generates a structured fatigue determination result according to the above rules, and outputs information including the current fatigue grade, the triggering rule, the judgment time and the like.

[0174] Preferably, the application also provides a long-haul task personnel operation fatigue detection system for performing the long-haul task personnel operation fatigue detection method as described above, and the long-haul task personnel operation fatigue detection system comprises:

[0175] A multi-source data acquisition module is configured to acquire personnel image data and personnel voice data.

[0176] A multi-modal feature recognition and anchor embedding module is configured to recognize a voice trigger event based on the personnel voice data to obtain voice trigger event data, extract facial dynamic features based on the personnel image data to obtain facial dynamic feature data, and perform cross-modal anchor embedding of the facial dynamic feature data on the voice trigger event data to obtain multi-modal event data.

[0177] A reaction trajectory modeling and delay calculation module is configured to calculate reaction delay based on the multi-modal event data to obtain event reaction data.

[0178] The attention evaluation and fatigue determination module is used for processing attention level according to the event reaction data to obtain event attention data, and determining fatigue according to the event attention data to obtain personnel fatigue data.

Claims

1. A method for detecting operator fatigue during long-range missions, characterized in that, Includes the following steps: Step S1: Acquire personnel image data and personnel voice data; Step S2: Recognize voice-triggered events based on personnel voice data to obtain voice-triggered event data; Facial dynamic features are extracted from personnel image data to obtain facial dynamic feature data; the facial dynamic feature data is then cross-modal embedded into voice-triggered event data to obtain multimodal event data. Step S3: Obtain the trigger anchor time from the multimodal event data to get the trigger anchor time data; The response behavior time data is captured by the trigger anchor time data to obtain the response behavior time data; the reaction time difference is calculated by the response behavior time data to obtain the reaction time difference data; and the reaction feature is extracted based on the reaction time difference data to obtain the event reaction data. Step S4: Process the attention level based on the event response data to obtain event attention data; determine fatigue based on the event attention data to obtain personnel fatigue data; The specific steps of reaction feature extraction are as follows: Reaction trajectory data is obtained by constructing reaction trajectory data based on reaction time difference data; phase space upsizing embedding is performed on reaction trajectory data to obtain reaction upsizing data; trend fluctuation dual-domain construction is performed based on reaction upsizing data to obtain dual-domain constructed data; trajectory geometric quantization is performed based on dual-domain constructed data to obtain trajectory geometric data. Change point detection is performed on the trajectory geometry data to obtain event response data.

2. The method according to claim 1, characterized in that, The personnel voice data includes the voice data of the first person and the voice data of the second person. Step S1 is as follows: Voice data of the first person and voice data of the second person were obtained by using in-cabin audio equipment and personal microphones. Personnel images are captured using in-cabin cameras to obtain personnel image data.

3. The method according to claim 2, characterized in that, Voice-triggered event recognition specifically refers to: Channel alignment is performed on the first person's voice data and the second person's voice data to obtain channel-aligned data; Cross-channel mutual verification is performed on the channel-aligned data to obtain cross-channel fused data; Two-layer keyword triggering is performed on cross-channel fused data to obtain trigger detection data; Voice trigger events are generated based on the trigger detection data to obtain voice trigger event data.

4. The method according to claim 3, characterized in that, Cross-channel mutual verification identification specifically includes: Perform single-channel speech recognition on the channel-aligned data to obtain speech recognition data; Consistency comparison is performed on the speech recognition data to obtain consistency comparison data; Based on the consistency comparison data, the difference location is obtained by locating the differences. Based on the differential positioning data, a confidence-weighted discrimination is performed to obtain weighted discrimination data; Re-discrimination is performed based on the weighted discrimination data to obtain re-discrimination data; Cross-channel fusion is performed on the re-discrimination data to obtain cross-channel fused data.

5. The method according to claim 3, characterized in that, The specific triggering of double-layer keywords is as follows: Acoustic layer candidate detection is performed on the cross-channel fused data to obtain acoustic layer candidate data; Text-level keyword detection is performed on cross-channel fused data to obtain text-level keyword data; Cross-layer consistency verification is performed on the candidate data of the acoustic layer and the keyword data of the text layer to obtain cross-layer consistent data; Trigger confidence assessment is performed on cross-layer consistency data to obtain trigger detection data.

6. The method according to claim 1, characterized in that, Facial dynamic feature extraction specifically involves: Face detection is performed based on personnel image data to obtain face detection data; Facial landmark recognition is performed on face detection data to obtain facial landmark data; Based on facial key point data, eye features and mouth features are extracted to obtain eye feature data and mouth feature data respectively. Cross-site correlation processing is performed based on eye feature data and mouth feature data to obtain cross-site correlation data; By performing collaborative pattern mapping on cross-site related data, dynamic facial feature data is obtained.

7. The method according to claim 1, characterized in that, Cross-modal anchoring specifically refers to: Voice event anchor points are extracted from the voice-triggered event data to obtain voice event anchor data; Facial feature fragment data is obtained by extracting facial feature fragments from voice event anchor data. Temporal alignment is performed based on voice event anchor data and facial feature fragment data to obtain temporal alignment data; Anchor point embedding is performed based on time-series aligned data to obtain multimodal event data.

8. The method according to claim 1, characterized in that, Step S4 is as follows: Attention level index is calculated based on event response data to obtain event attention data; Short-term trend aggregation is performed on event attention data to obtain attention trend feature data; Fatigue is assessed based on event attention data and attention trend characteristics data to obtain personnel fatigue data.

9. A fatigue detection system for personnel on long-range missions, characterized in that, For performing the operational fatigue detection method for long-distance mission personnel as described in claim 1, the operational fatigue detection system for long-distance mission personnel includes: The multi-source data acquisition module is used to acquire personnel image data and personnel voice data; The multimodal feature recognition and embedding module is used to recognize voice-triggered events based on human voice data to obtain voice-triggered event data; extract facial dynamic features based on human image data to obtain facial dynamic feature data; and embed the facial dynamic feature data into the voice-triggered event data across modalities to obtain multimodal event data. The reaction trajectory modeling and delay calculation module is used to perform reaction delay calculations on multimodal event data to obtain event reaction data; The attention assessment and fatigue determination module is used to process attention levels based on event response data to obtain event attention data; and to determine fatigue based on event attention data to obtain personnel fatigue data.

Images