Multi-modal fusion fatigue screen monitoring identification and reminding method and system

Abstract

The invention provides a multi-modal fusion fatigue screen monitoring recognition and reminding method and system, and the method comprises the steps: data collection: collecting the visual data of a screen monitoring person in real time, and synchronously collecting the physiological data; multi-modal data fusion: adopting a feature weighted fusion method based on an entropy weight method to adaptively distribute weights and generate a fusion fatigue index FFI by quantifying dynamic information entropy of each visual data and physiological data; fatigue grade classification: realizing three-level fatigue state judgment based on a fusion fatigue index FFI obtained by multi-modal data fusion and a dynamic decision tree model; and dynamic intervention: based on a fatigue grade classification result, adopting a double-channel intervention mechanism of bracelet touch alarm and automatic telephone call value length. When fatigue and inattention of a monitoring screen watchman occur, the monitoring screen watchman can be timely and accurately identified and a reminding intervention mechanism is started, so that the continuity and the safety of the monitoring screen work are ensured, and bad safety production events caused by human reasons are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent monitoring of power systems, in particular to a multi-modal fusion fatigue monitoring screen identification and reminding method and system. BACKGROUND

[0002] In the power production industry, the monitoring screen work in the power plant central control room mainly relies on manual monitoring and regular inspection. The monitoring screen operator needs to face the screen and monitor the signals reported by the equipment for 24 hours without interruption and make correct judgments. Long-time monitoring screen operation (especially at night) can easily cause fatigue of the operator, which affects work efficiency and safety production. However, there is currently a lack of intelligent fatigue monitoring screen identification and reminding means in the power production industry.

[0003] In the prior art, some industries use simple camera face recognition (eyelid closure frequency) to detect fatigue, but these solutions have high false positive rates, delayed responses, and low integration with work scenarios. For the power industry, which has extremely high requirements for safety and stability, the prior art cannot meet the precise fatigue identification needs of 24-hour continuous monitoring screen. SUMMARY

[0004] To solve the problem of fatigue of monitoring screen personnel at night in the power plant central control room, the present application provides a multi-modal fusion fatigue monitoring screen identification and reminding method and system, which aims to accurately identify and start the reminding intervention mechanism when the monitoring screen operator is fatigued and not focused, to ensure the continuity and safety of the monitoring screen work and avoid adverse safety production events caused by human factors.

[0005] To achieve the above technical features, the purpose of the present application is achieved as follows: a multi-modal fusion fatigue monitoring screen identification and reminding method, the method comprising: Data acquisition: by setting a high-precision infrared camera in the monitoring screen equipment area, real-time visual data of the monitoring screen personnel is collected, and by wearing a smart bracelet on the wrist of the monitoring screen personnel, physiological data is synchronously collected; Multi-modal data fusion: a feature weighting fusion method based on entropy weight method is adopted, the dynamic information entropy of each visual data and physiological data is quantified, the weight is adaptively allocated and the fusion fatigue index is generated ; Fatigue level classification: based on the fusion fatigue index obtained by multi-modal data fusion and a dynamic decision tree model, three-level fatigue state determination is achieved; Dynamic intervention: based on the fatigue level classification result, a dual-channel intervention mechanism of bracelet tactile alarm and automatic telephone call to the operator is adopted.

[0006] Preferably, the high-precision infrared camera is positioned directly above the monitoring device, so that the high-precision infrared camera vertically covers the facial area of ​​the monitoring personnel; The smart bracelet is a medical-grade smart bracelet with a sampling rate greater than 100Hz and is located on the wrist of the monitoring personnel.

[0007] Preferably, the visual data includes: eye closure frequency. P Pupil diameter change rate ∆D and frequency of yawning Y ; The frequency of eye closure P Is adopted The algorithm locates the eye region and calculates the percentage of time the eyelids are closed per unit of time, i.e., the eye closure frequency. = (Number of frames with eyes closed / Total number of frames) × 100%; The rate of change in pupil diameter It uses an ellipse fitting algorithm to track the pupil contour and calculate the real-time pupil diameter. _ Real-time relative to baseline value _ The dynamic fluctuation amplitude of the baseline, i.e., the rate of change in pupil diameter. = ( _ real time- _ Baseline) / _ Baseline × 100%; The frequency of yawning Y During the data collection process, a yawn with a jaw opening greater than 15mm and a duration greater than 2 seconds was defined as a valid yawn. Continuous movements with a mouth opening duration of ≥2 seconds were identified through facial key point detection, and the number of yawns per minute was counted.

[0008] Preferably, the physiological data includes: heart rate variability. H and blood oxygen saturation S ; Heart rate variability H It involves acquiring the RR interval sequence using a photoplethysmography sensor and extracting time-domain indices. and frequency domain indicators ratio; blood oxygen saturation S It is based on the dual-wavelength transmission measurement method to calculate the percentage of arterial blood oxygen saturation.

[0009] Preferably, the feature weighting fusion method based on entropy weight method is as follows: 1) Data preprocessing and feature standardization: Visual feature sequence: eye closure frequency P , pupil diameter change rate , yawn frequency Y Sliding window analysis is performed respectively, window length T seconds, step t seconds, and the mean and standard deviation P of eye closure frequency in each window are extracted; ; Maximum fluctuation amplitude of pupil diameter ; Yawn count cumulative value ; Physiological feature sequence: heart rate variability , ratio and blood oxygen saturation S are divided into synchronous windows, and the following are calculated: Heart rate variability time-frequency joint index: ; Blood oxygen saturation decline slope: ; Normalization: Min-Max normalization is used to map each feature to the interval [0, 1] to eliminate dimensional differences: ; 2) Dynamic weight distribution based on entropy weight method: Information entropy calculation: The information entropy value of each window is calculated for 5 types of features ( , , , , ): ; Where, is the number of samples in the window, is the normalized proportion of the th feature in the th sample; Weight generation: According to the entropy value, the weight of each feature is determined. The smaller the entropy value, the higher the index discrimination, and the greater the weight: ; 3) Fatigue index fusion and output: Weighted fusion formula: The normalized feature value is combined with the weight to generate a fusion fatigue index : ; wherein, is the normalized value of the th feature.

[0010] Preferably, the fatigue index obtained based on the multi-modal data fusion is combined with a dynamic decision tree model to realize the specific steps of three-level fatigue state determination as follows: 1) Fatigue grade division standard: Normal state is : and meets all the following conditions: ≤15%; Oxygen saturation S fluctuation range≤±2%; Pupil diameter change rate ≤10%.

[0011] Mild fatigue is : or meets any two of the following: >15% and lasts≥30 seconds; ≥-0.2% / min; Decreased by≥25% of the baseline value; Severe fatigue is : ≥0.6 or meets any one of the following: Detect yawning frequency in 3 consecutive windows ≥3 times / min; Pupil diameter continues to dilate for more than 1 minute; Oxygen saturation S absolute value<94% and lasts≥20 seconds; 2) Classification algorithm implementation: Input data: Input the value obtained by multi-modal data fusion; original feature parameters ( , , , , ); Dynamic decision tree model: The hierarchical decision logic is constructed using a dynamic decision tree model based on the C4.5 algorithm, with the following priority order: Level 1 Judgment: Inspection Trigger? If the threshold is ≥0.6, the classifier will classify the classifier directly. Secondary judgment: If Not reached Then, rule matching is performed based on the original features, including yawning frequency and blood oxygen saturation. Level 3 Judgment: Correct For samples with critical values ​​of 0.3-0.35 or 0.55-0.6, a support vector machine is introduced for secondary verification.

[0012] Preferably, the kernel function selected for the support vector machine in the three-level decision is the radial basis function, with hyperparameters C=1.0 and γ=0.01.

[0013] Preferably, the specific steps of the dual-channel intervention mechanism based on fatigue level classification results, employing wristband tactile alarm and automatic telephone call value, are as follows: 1) Intervention strategies and implementation: normal state : No intervention measures: The bracelet remains in silent mode, with the LED indicator showing a solid green light; physiological data is continuously recorded in the background without triggering any external communication. Mild fatigue : Response to Level 1 Intervention: Tactile alert: The wristband's linear motor triggers short pulse vibrations; Voice prompt: Play the voice message "Fatigue signs detected, please focus on monitoring" through the bracelet speaker (volume 40dB, played twice).

[0014] Severe fatigue : Response to Level 2 intervention: High-intensity tactile alarm: The bracelet switches to emergency vibration mode; Voice alarm: The voice message "Stop work immediately! Calling the shift supervisor!" plays repeatedly. Telephone calling system: Automatically dials preset long-length mobile phone numbers through an integrated module; After the call is established, a synthesized voice alarm is played: "Warning! Warning! The operator monitoring screen staff is in a state of severe fatigue. Please intervene immediately!" 2) Dynamic Priority and Conflict Resolution: Intervention and control: Telephone calls have the highest priority. Once triggered, they forcibly occupy the communication channel and suspend other data uploads. If the shift supervisor answers and confirms, the wristband vibration will stop immediately, and the voice will switch to "Shift supervisor has responded, please wait for instructions." Manual confirmation feedback: During a call, the shift leader can press "#1" to confirm receiving the alarm. The system will record the operation time and the leader's identity information. If no confirmation is made, the call will continue until the preset timeout, after which the emergency dispatch process will be initiated.

[0015] Preferably, the short pulse vibration frequency is 75Hz, the amplitude is 0.5G, the duration is 2 seconds × 2 times, and the interval is 3 seconds. The voice prompt volume is 40dB and is broadcast twice. The emergency vibration mode has a frequency of 150Hz, an amplitude of 2.0G, and continuous vibration without interruption. The voice alarm volume is 65dB and it repeats continuously.

[0016] Another aspect of the present invention provides a multimodal fusion fatigue monitoring recognition and alert system, the system being used to implement a multimodal fusion fatigue monitoring recognition and alert method, comprising: Data acquisition module: Collects visual and physiological data from monitoring personnel; Data fusion analysis module: It adopts a feature weighted fusion method based on entropy weighting, which quantifies the dynamic information entropy of each physiological and visual indicator, adaptively allocates weights, and generates a fusion fatigue index. Fatigue level classification module: Based on multimodal fusion fatigue index and dynamic decision tree model, it realizes the determination of three-level fatigue state; Dynamic intervention module: Based on the fatigue level classification results, a dual-channel intervention mechanism is implemented by using wristband tactile alarm and automatic telephone call value.

[0017] The present invention has the following beneficial effects: 1. This invention provides a multimodal fusion fatigue monitoring identification and reminder method and system, which can promptly and accurately identify and activate the reminder intervention mechanism when the monitoring duty officer is fatigued or inattentive, so as to ensure the continuity and safety of monitoring work and avoid adverse safety production incidents caused by human factors.

[0018] 2. This invention proposes a multimodal fusion fatigue monitoring identification and reminder method, filling the gap in the power production industry's monitoring work which lacks a real-time fatigue monitoring and reminder mechanism.

[0019] 3. This invention solves the problem of insufficient accuracy in fatigue identification by a single sensor. By using the collaborative judgment of a camera, biosensors, and operational behavior analysis, it ensures that fatigue status can be identified in a timely and accurate manner and intervention measures can be taken. 4. This invention establishes a linkage early warning and intervention mechanism with the power monitoring system, classifies the identified fatigue state, and initiates different levels of response.

[0020] 5. This invention is applicable to working scenarios such as hydropower stations and thermal power plants that require long-term monitoring. Attached Figure Description

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Figure 1 This is a schematic diagram of the overall framework of a multimodal fusion fatigue monitoring recognition and alert method and system according to the present invention. Detailed Implementation

[0023] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0024] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Basic principle of this invention: The core principle of this invention is to collect visual and physiological data through an infrared camera and a smart bracelet, respectively, and combine a dynamic weighted fusion algorithm and a graded intervention strategy to achieve high-precision fatigue monitoring and alert intervention. Specifically, the visual module extracts eye closure frequency, dynamic fluctuations in pupil diameter, and yawning frequency in real time, while the physiological module monitors heart rate variability and blood oxygen saturation synchronously through a medical-grade bracelet. An entropy weighting method is used to dynamically fuse multimodal data, assigning weights based on entropy values ​​to quantify their contribution to fatigue characterization, and generating a fused fatigue index (FFI). Based on the FFI, fatigue states are divided into three levels: normal state (…). FFI <0.3), mild fatigue (0.3≤ FFI <0.6) and severe fatigue ( FFI (≥0.6), and then different levels of intervention measures are initiated to ensure the continuity and safety of the monitoring work and to avoid adverse safety production incidents caused by human factors.

[0026] Example 1: like Figure 1 As shown, a multimodal fusion fatigue monitoring recognition and alert method is provided, the method comprising: I. Data Collection: The data acquisition process collects visual and physiological data. A high-precision infrared camera is installed in the monitoring area to collect visual data from the monitoring personnel in real time, and a smart bracelet worn on the wrist of the monitoring personnel simultaneously collects physiological data. In this embodiment, a high-precision infrared camera is positioned directly above the monitoring device, vertically covering the face area of ​​the monitoring personnel, to ensure the accuracy and comprehensiveness of data collection. Visual data includes: eye closure frequency P Pupil diameter change rate and frequency of yawning Y ; Eye closure frequency ( ， use (Indicated): Under fatigue, the duty officer's eyelid closure time increased significantly, and the closing speed slowed down. The algorithm locates the eye region and calculates the percentage of time the eyelids are closed per unit of time, i.e., the eye closure frequency. = (Number of frames with eyes closed / Total number of frames) × 100%; Pupil diameter change rate When fatigued, the pupillary light reflex of the duty officer weakens, the frequency of pupil diameter fluctuations decreases, and the amplitude of diameter changes decreases. An ellipse fitting algorithm is used to track the pupil contour and calculate the real-time pupil diameter. _ Real-time relative to baseline value _ The dynamic fluctuation amplitude of the baseline, i.e., the rate of change in pupil diameter. = ( _ real time- _ Baseline) / _ Baseline × 100%; yawning frequency Y Yawning is a physiological response to brain hypoxia, but frequent yawning (>3 times / 10 minutes) is highly correlated with central fatigue. A valid yawn is defined as a jaw opening greater than 15mm and a duration greater than 2 seconds. Continuous movements with a mouth opening duration ≥2 seconds are identified using facial key point detection, and the number of yawns per minute is counted.

[0027] The following physiological data were simultaneously collected by wearing a medical-grade smart bracelet with a sampling rate greater than 100Hz on the wrist of the duty officer: Heart rate variability ( ,use (Note: When fatigued, the on-duty staff experience increased sympathetic nerve excitability and parasympathetic nerve inhibition, leading to heart rate variability.) Overall reduction. RR interval sequences were acquired using a PPG (photoplethysmography) sensor, and time-domain indices were extracted. (Standard deviation of normal sinus interval) and frequency domain indices Ratio (low-frequency / high-frequency power ratio); Blood oxygen saturation ( ,use (Representation): Based on the dual-wavelength transmission measurement method, the percentage of arterial blood oxygen saturation is calculated.

[0028] II. Multimodal data fusion: An entropy-weighted feature-weighted fusion method is adopted. By quantifying the dynamic information entropy of each visual and physiological data, weights are adaptively assigned and a fusion fatigue index is generated. The specific implementation steps are as follows: 1) Data preprocessing and feature standardization: Visual feature sequence: frequency of eye closure P Pupil diameter change rate yawning frequency Y Perform sliding window analysis separately, with a window length of T seconds and a step size of t seconds, and extract the contents of each window: Eye closing frequency P mean with standard deviation ; Maximum fluctuation range of pupil diameter ; Cumulative number of yawns ; Physiological characteristic sequence: response to heart rate variability , Ratio and blood oxygen saturation S Perform synchronous window segmentation and calculation: Heart rate variability time-frequency combined index: ; Slope of decrease in blood oxygen saturation: ; Normalization: Min-Max standardization is used to map each feature to the [0,1] interval to eliminate dimensional differences. ; 2) Dynamic weight allocation based on entropy weight method: Information entropy calculation: For 5 types of features ( , , , , Calculate the information entropy value within each window: ; in, The number of samples within the window. For the first The feature in the first The normalized proportion of the sample; Weight generation: The weights of each feature are determined based on its entropy value; the smaller the entropy value, the higher the discriminative power of the indicator, and the greater the weight. ; 3) Fatigue index fusion and output: Weighted fusion formula: The standardized feature values ​​are combined with the weights to generate the fusion fatigue index. : ; in, For the first The normalized values ​​of each feature.

[0029] III. Fatigue Level Classification: The fusion fatigue index obtained from multimodal data fusion Using a dynamic decision tree model, a three-level fatigue state determination is achieved. The specific steps are as follows: 1) Fatigue level classification standard: (Normal state): And all of the following conditions must be met: ≤15%; blood oxygen saturation S Fluctuation range ≤ ±2%; Pupil diameter change rate ≤10%.

[0030] (Mild fatigue): Or satisfy any two of the following: >15% and lasting ≥30 seconds; ≥-0.2% / minute; HRV Composite Index A decrease of ≥25% from the baseline value; (Severe fatigue): ≥0.6 or satisfy any of the following: A yawning frequency of ≥3 times / minute was detected in 3 consecutive windows (≥30 seconds); The pupil diameter continues to dilate ( >20%) for more than 1 minute; blood oxygen saturation S The absolute value is less than 94% and lasts for ≥20 seconds; 2) Classification algorithm implementation: Input data: Input multimodal data fusion Values; original feature parameters ( , , , , ); Dynamic decision tree model: The hierarchical decision logic is constructed using a dynamic decision tree model based on the C4.5 algorithm, with the following priority order: Level 1 Judgment: Inspection Trigger? If the threshold is ≥0.6, the classifier will classify the classifier directly. Secondary judgment: If Not reached Then, rule matching is performed based on the original features, including yawning frequency and blood oxygen saturation. Level 3 Judgment: Correct For samples with critical values ​​of 0.3-0.35 or 0.55-0.6, a support vector machine is introduced for secondary verification.

[0031] Preferably, the kernel function selected for the support vector machine in the three-level decision is the radial basis function, with hyperparameters C=1.0 and γ=0.01.

[0032] IV. Dynamic Intervention: Based on the fatigue level classification results, a dual-channel intervention mechanism combining wristband tactile alarm and automatic telephone call threshold is adopted. The specific steps are as follows: 1) Intervention strategies and implementation: (Normal state): No intervention measures: The bracelet remains in silent mode, with the LED indicator showing a solid green light (brightness 30 cd / m²); physiological data is continuously recorded in the background without triggering any external communication. (Mild fatigue): Response to Level 1 Intervention: Tactile cues: The band's linear motor (LRA) triggers short pulse vibrations (frequency 75Hz, amplitude 0.5G, lasting 2 seconds x 2 times, with a 3-second interval); Voice prompt: Play the voice message "Fatigue signs detected, please focus on monitoring" through the bracelet speaker (volume 40dB, played twice).

[0033] (Severe fatigue): Response to Level 2 intervention: High-intensity tactile alarm: The bracelet switches to emergency vibration mode (frequency 150Hz, amplitude 2.0G, continuous vibration without interruption). Voice alarm: The voice message "Stop work immediately! Calling the shift leader!" plays repeatedly (volume 65dB). Telephone calling system: Automatically dials preset long-length mobile phone numbers through an integrated module; After the call is established, a synthesized voice alarm is played: "Warning! Warning! The operator monitoring screen staff is in a state of severe fatigue. Please intervene immediately!"

[0034] 2) Dynamic Priority and Conflict Resolution: Intervention and control: Telephone calls have the highest priority. Once triggered, they forcibly occupy the communication channel and suspend other data uploads. If the shift supervisor answers and confirms, the wristband vibration will stop immediately, and the voice will switch to "Shift supervisor has responded, please wait for instructions." Manual confirmation feedback: During a call, the shift leader presses "#1" to confirm receiving the alarm, and the system records the operation time and identity information; if no confirmation is received, the call continues until the preset timeout (120 seconds), after which the emergency dispatch process is initiated.

[0035] Example 2: Another aspect of the present invention provides a multimodal fusion fatigue monitoring recognition and alert system, comprising: Data acquisition module: Collects visual and physiological data from monitoring personnel; Data fusion analysis module: It adopts a feature weighted fusion method based on entropy weighting, which quantifies the dynamic information entropy of each physiological and visual indicator, adaptively allocates weights, and generates a fusion fatigue index. Fatigue level classification module: Based on multimodal fusion fatigue index and dynamic decision tree model, it realizes the determination of three-level fatigue state; Dynamic intervention module: Based on the fatigue level classification results, a dual-channel intervention mechanism is implemented by using wristband tactile alarm and automatic telephone call value.

[0036] Through the aforementioned system, during operation, the data acquisition module extracts real-time data on eye closure frequency, dynamic fluctuations in pupil diameter, and yawning frequency. A medical-grade wristband simultaneously monitors heart rate variability and blood oxygen saturation. The entropy weighting method is used to dynamically fuse multimodal data, assigning weights based on entropy values ​​to quantify their contribution to fatigue assessment and generating a fused fatigue index (FFI). Based on the FFI, fatigue states are categorized into three levels: normal (FFI < 0.3), mild fatigue (0.3 ≤ FFI < 0.6), and severe fatigue (FFI ≥ 0.6). Different levels of intervention measures are then initiated for each level to ensure the continuity and safety of the monitoring work and prevent adverse safety incidents caused by human error.

[0037] Example 3: Taking the operation of a duty officer at a hydropower station as an example: Input data: ; mean yawning frequency Y = 2 times / minute; (Lasts 25 seconds).

[0038] Classification process: Level 1 judgment: No trigger ; Level 2 judgment: examine And lasting for 25 seconds, satisfying the requirement. of Absolute value condition; The frequency of yawning did not reach the threshold. Not continuously exceeding the limit; Level 3 Judgment: Because Trigger a single Conditions for starting SVM verification: SVM input features: [ , , = [0.58, 0.18, 93.5]; Output probability: Confidence level: 82.7%; Final judgment: (Severe fatigue) triggers secondary intervention measures.

[0039] Secondary intervention initiated: 1. The wristband vibrates continuously at high intensity and plays alarm voice messages simultaneously; the integrated module dials the shift supervisor's mobile phone.

[0040] 2. Call connection and alarm: After the shift leader answered the call, the system announced: "Warning! Warning! Operator 001's monitoring screen shows the shift member is in a state of severe fatigue. Please intervene immediately!" Press and hold the "#1" key to confirm, and the system will record the response time.

[0041] 3. Closed-loop management: The wristband stopped vibrating, and a voice prompt appeared saying, "Shift leader has confirmed, please wait for instructions." The present invention and its embodiments have been described above. This description is not restrictive. The accompanying drawings are only one embodiment of the present invention. The actual structure is not limited to this. In short, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the present invention, such design should fall within the protection scope of the present invention.

Claims

1. A multimodal fusion method for fatigue monitoring, identification, and alerting, characterized in that, The method includes: Data Acquisition: By setting up high-precision infrared cameras in the monitoring equipment area, visual data of monitoring personnel is collected in real time, and physiological data is collected simultaneously by wearing smart bracelets on the wrists of monitoring personnel; Multimodal data fusion: A feature-weighted fusion method based on entropy weighting is adopted. By quantifying the dynamic information entropy of each visual and physiological data, weights are adaptively allocated and a fusion fatigue index is generated. ; Fatigue level classification: Fusion fatigue index obtained from multimodal data fusion Using a dynamic decision tree model, a three-level fatigue state determination is achieved; Dynamic intervention: Based on the fatigue level classification results, a dual-channel intervention mechanism of wristband tactile alarm and automatic telephone call value is adopted.

2. The fatigue monitoring and alerting method based on multimodal fusion according to claim 1, characterized in that, The high-precision infrared camera is positioned directly above the monitoring device, ensuring that it vertically covers the facial area of ​​the person monitoring the device. The smart bracelet is a medical-grade smart bracelet with a sampling rate greater than 100Hz and is located on the wrist of the monitoring personnel.

3. The fatigue monitoring and alerting method based on multimodal fusion according to claim 1, characterized in that, The visual data includes: eye closure frequency. P Pupil diameter change rate and frequency of yawning Y ; The frequency of eye closure P Is adopted The algorithm locates the eye region and calculates the percentage of time the eyelids are closed per unit of time, i.e., the eye closure frequency. = (Number of frames with eyes closed / Total number of frames) × 100%; The rate of change in pupil diameter It uses an ellipse fitting algorithm to track the pupil contour and calculate the real-time pupil diameter. _ Real-time relative to baseline value _ The dynamic fluctuation amplitude of the baseline, i.e., the rate of change in pupil diameter. = ( _ real time- _ Baseline) / _ Baseline × 100%; The frequency of yawning Y During the data collection process, a yawn with a jaw opening greater than 15mm and a duration greater than 2 seconds was defined as a valid yawn. Continuous movements with a mouth opening duration of ≥2 seconds were identified through facial key point detection, and the number of yawns per minute was counted.

4. The fatigue monitoring and alerting method based on multimodal fusion according to claim 1, characterized in that, The physiological data includes: heart rate variability. H and blood oxygen saturation S ; Heart rate variability H It involves acquiring the RR interval sequence using a photoplethysmography sensor and extracting time-domain indices. and frequency domain indicators ratio; blood oxygen saturation S It is based on the dual-wavelength transmission measurement method to calculate the percentage of arterial blood oxygen saturation.

5. The fatigue monitoring and alerting method based on multimodal fusion according to claim 4, characterized in that, The specific steps of the feature weighted fusion method based on entropy weighting, which quantifies the dynamic information entropy of each visual and physiological data, adaptively allocates weights, and generates a fusion fatigue index, are as follows: 1) Data preprocessing and feature standardization: Visual feature sequence: frequency of eye closure P Pupil diameter change rate yawning frequency Y Perform sliding window analysis separately, with a window length of T seconds and a step size of t seconds, and extract the contents of each window: Eye closing frequency P mean with standard deviation ; Maximum fluctuation range of pupil diameter ; Cumulative number of yawns ; Physiological characteristic sequence: response to heart rate variability , Ratio and blood oxygen saturation S Perform synchronous window segmentation and calculation: Heart rate variability time-frequency combined index: ; Slope of decrease in blood oxygen saturation: ; Normalization: Min-Max standardization is used to map each feature to the [0,1] interval to eliminate dimensional differences. ； 2) Dynamic weight allocation based on entropy weight method: Information entropy calculation: For 5 types of features ( , , , , Calculate the information entropy value within each window: ； in, The number of samples within the window. For the first The feature in the first The normalized proportion of the sample; Weight generation: The weights of each feature are determined based on its entropy value; the smaller the entropy value, the higher the discriminative power of the indicator, and the greater the weight. ； 3) Fatigue index fusion and output: Weighted fusion formula: The standardized feature values ​​are combined with the weights to generate the fusion fatigue index. : ； in, For the first The normalized values ​​of each feature.

6. The fatigue monitoring and alerting method based on multimodal fusion according to claim 5, characterized in that, The fatigue index obtained based on multimodal data fusion The specific steps for determining the three-level fatigue state using the dynamic decision tree model are as follows: 1) Fatigue level classification standard: normal state : And all of the following conditions must be met: ≤15%； blood oxygen saturation S Fluctuation range ≤ ±2%; Pupil diameter change rate ≤10%; Mild fatigue : Or satisfy any two of the following: >15% and lasting ≥30 seconds; ≥-0.2% / minute; A decrease of ≥25% from the baseline value; Severe fatigue : ≥0.6 or satisfy any of the following: Three consecutive windows detected yawning frequency ≥3 times / minute; The pupil diameter remained dilated for more than 1 minute; blood oxygen saturation S The absolute value is less than 94% and lasts for ≥20 seconds; 2) Classification algorithm implementation: Input data: Input multimodal data fusion Values; original feature parameters ( , , , , ); Dynamic decision tree model: The hierarchical decision logic is constructed using a dynamic decision tree model based on the C4.5 algorithm, with the following priority order: Level 1 Judgment: Inspection Trigger? If the threshold is ≥0.6, the classifier will classify the classifier directly. Secondary judgment: If Not reached Then, rule matching is performed based on the original features, including yawning frequency and blood oxygen saturation. Level 3 Judgment: Correct For samples with critical values ​​of 0.3-0.35 or 0.55-0.6, a support vector machine is introduced for secondary verification.

7. The fatigue monitoring and alerting method based on multimodal fusion according to claim 6, characterized in that, In the three-level decision-making process, the kernel function selected for the support vector machine is the radial basis function, with hyperparameters C=1.0 and γ=0.

01.

8. The fatigue monitoring and alerting method based on multimodal fusion according to claim 6, characterized in that, The specific steps of the dual-channel intervention mechanism based on fatigue level classification results and combining wristband tactile alarm and automatic telephone call threshold are as follows: 1) Intervention strategies and implementation: normal state : No intervention measures: The bracelet remains in silent mode, with the LED indicator showing a solid green light; physiological data is continuously recorded in the background without triggering any external communication. Mild fatigue : Response to Level 1 Intervention: Tactile alert: The wristband's linear motor triggers short pulse vibrations; Voice prompt: Play the voice message "Fatigue signs detected, please focus on monitoring" through the wristband speaker (volume 40dB, played twice). Severe fatigue : Response to Level 2 intervention: High-intensity tactile alarm: The bracelet switches to emergency vibration mode; Voice alarm: The voice message "Stop work immediately! Calling the shift supervisor!" plays repeatedly. Telephone calling system: Automatically dials preset long-length mobile phone numbers through an integrated module; After the call is established, a synthesized voice alarm is played: "Warning! Warning! The operator monitoring screen staff is in a state of severe fatigue. Please intervene immediately!" 2) Dynamic Priority and Conflict Resolution: Intervention and control: Telephone calls have the highest priority. Once triggered, they forcibly occupy the communication channel and suspend other data uploads. If the shift supervisor answers and confirms, the wristband vibration will stop immediately, and the voice will switch to "Shift supervisor has responded, please wait for instructions." Manual confirmation feedback: During a call, the shift leader can press "#1" to confirm receiving the alarm. The system will record the operation time and the leader's identity information. If no confirmation is made, the call will continue until the preset timeout, after which the emergency dispatch process will be initiated.

9. The fatigue monitoring and alerting method based on multimodal fusion according to claim 8, characterized in that, The short pulse vibration frequency is 75Hz, the amplitude is 0.5G, and it lasts for 2 seconds × 2 times with a 3-second interval. The voice prompt volume is 40dB and is broadcast twice. The emergency vibration mode has a frequency of 150Hz, an amplitude of 2.0G, and continuous vibration without interruption. The voice alarm volume is 65dB and it repeats continuously.

10. A multimodal fusion fatigue monitoring, recognition, and alerting system, characterized in that, The system is used to implement the multimodal fusion fatigue monitoring recognition and reminder method according to any one of claims 1-9, including: Data acquisition module: Collects visual and physiological data from monitoring personnel; Data fusion analysis module: It adopts a feature weighted fusion method based on entropy weighting, which quantifies the dynamic information entropy of each physiological and visual indicator, adaptively allocates weights, and generates a fusion fatigue index. Fatigue level classification module: Based on multimodal fusion fatigue index and dynamic decision tree model, it realizes the determination of three-level fatigue state; Dynamic intervention module: Based on the fatigue level classification results, a dual-channel intervention mechanism is implemented by using wristband tactile alarm and automatic telephone call value.

Images