Intelligent monitoring system for states of workers in power industry based on multi-modal monitoring

Abstract

The invention discloses an intelligent monitoring system for the state of workers in the power industry based on multi-modal monitoring, and relates to the technical field of state monitoring. The invention aims to solve the problem that an alarm cannot be accurately given out due to low monitoring accuracy and efficiency of an existing power industry personnel state monitoring method. The method comprises the following steps: acquiring a face video and a body video of a worker, and acquiring the face video and the body video; obtaining a physiological index value of the worker; the physiological index values comprise heartbeat information, blood pressure information and oxyhemoglobin saturation; obtaining a state index value of the worker according to the face video, the body video and the physiological index value of the worker; and judging the state of the current worker according to the worker state index value and giving an alarm. The method is used for monitoring the state of the staff in the power industry.

Description

Technical Field

[0001] The present invention relates to the technical field of status monitoring, and in particular to an intelligent status monitoring system for power industry workers based on multi-modal monitoring. Background Art

[0002] In the power industry, the status of staff plays a vital role in ensuring the safe and stable operation of the power grid. As a basic industry of the national economy, the power industry's work characteristics often require personnel to maintain a high degree of concentration and accurate judgment for a long time. Whether it is power generation, transmission or distribution, any operational errors caused by fatigue, distraction or lack of concentration may cause serious power accidents, posing a huge threat to personal safety, equipment safety and power supply reliability. Therefore, it is very important for the power industry to monitor the status of staff.

[0003] At present, the main methods used by the power industry for personnel status monitoring include manual observation (executed by the duty supervisor and guardian) and simple timer reminder methods. Although manual observation can directly observe some obvious behavioral changes of personnel, it is difficult to continuously, comprehensively and accurately monitor the status of each staff member due to the limitations of human attention and observation ability. At the same time, manual observation has the disadvantages of strong subjectivity, easy fatigue, slow reaction speed, etc., which is difficult to adapt to the modern power industry's needs for safe, efficient and accurate monitoring. Although simple timer reminders can remind staff to take a rest or adjust their status to a certain extent, they lack consideration of the differences between different individuals, different working environments and different work tasks, so the accuracy and effectiveness of reminders are often limited. Therefore, the existing personnel status monitoring methods still have the problems of low monitoring accuracy and low efficiency, which leads to the inability to accurately issue alarms. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the existing power industry personnel status monitoring methods have low monitoring accuracy and efficiency, resulting in the inability to accurately issue alarms, and propose an intelligent monitoring system for the status of power industry personnel based on multimodal monitoring.

[0005] An intelligent monitoring system for the status of power industry workers based on multimodal monitoring, comprising: a camera module, a contact sensor module, a status analysis module, and an alarm module;

[0006] The camera module is used to obtain facial video and body video of the staff member, and send the facial video and body video to the status analysis module;

[0007] The contact sensor module is used to obtain the physiological index values ​​of the staff and send the physiological index values ​​of the staff to the state analysis module;

[0008] The physiological index values ​​include: heart rate information, blood pressure information, and blood oxygen saturation;

[0009] The state analysis module is used to obtain the staff state index value based on the staff's facial video, body video and physiological index value, and send the staff state index value to the alarm module;

[0010] The alarm module determines the current status of the staff member according to the staff member status index value and issues an alarm.

[0011] Furthermore, the state analysis module is used to obtain the staff state index value according to the staff's facial video, body video and physiological index value, and send the staff state index value to the alarm module, specifically:

[0012] Step 1: extract each frame of the staff member's facial video and body video, pre-process each frame, and extract the staff member's facial features and body features;

[0013] Step 2: Using the facial features and facial video of the staff member, obtain the sleepiness index value corresponding to the facial features of the staff member;

[0014] Step 3: Using the physical characteristics and body video of the staff, obtain the sleepiness index value corresponding to the physical characteristics of the staff;

[0015] Step 4: Using the physical characteristics and body video of the staff member, obtain the health status index value corresponding to the physical characteristics of the staff member;

[0016] Step 5: Using the physiological index values ​​of the staff, obtain the health status index values ​​corresponding to the physiological index values;

[0017] Step 6: Send the sleepiness index value corresponding to the facial features of the staff member, the sleepiness index value corresponding to the physical features of the staff member, the health status index value corresponding to the physical features of the staff member, and the health status index value corresponding to the physiological index value to the alarm module.

[0018] Furthermore, the step 1 extracts each frame of the facial video and body video of the staff member, pre-processes each frame, and extracts the facial features and body features of the staff member, specifically:

[0019] First, extract each frame of the staff member's facial video and body video, and convert each facial image and body image into a grayscale image;

[0020] Then, facial grayscale images and facial videos are used to obtain the facial features of the staff;

[0021] Facial features include: distance between upper eyelid and lower eyelid, pupil diameter, eyebrow position, head position, eye closure time, blinking frequency per unit time;

[0022] Finally, the body features are obtained using the body grayscale image;

[0023] The body features include: torso position, hand position, and head position.

[0024] Furthermore, the sleepiness index value corresponding to the facial features of the staff member is obtained by using the facial features and facial video of the staff member in step 2, specifically:

[0025] Step 21: Use the facial features of the staff to obtain the eye feature values ​​of the staff, specifically:

[0026] Step 2: Using the distance between the upper eyelid and the lower eyelid of worker i in the t-th frame facial image, obtain the eye opening amplitude feature value of worker i in the t-th frame facial image, specifically:

[0027] A1. Obtain the vertical distance H between the upper eyelid edge and the lower eyelid edge of worker i in the t-th frame facial image. i-t ;

[0028] A2. Get the preset eye-opening distance of worker i:

[0029] Obtain the vertical distance between the upper eyelid edge and the lower eyelid edge when the worker i works normally and is not sleepy during the preset first time period, and obtain the average value of all vertical distances between the upper eyelid edge and the lower eyelid edge as the preset eye opening distance of the worker i;

[0030] A3. H i-t Compared with the preset eye opening distance of 0.4 times, if H i-t If the eye opening distance is greater than 0.4 times the preset eye opening distance, the eye opening amplitude characteristic value is set to 0; if H i-t If the eye opening distance is less than or equal to 0.4 times the preset eye opening distance, the eye opening amplitude characteristic value is set to 1;

[0031] Step 212: Use the pupil diameter of worker i in the t-th frame facial image to obtain the pupil size feature value:

[0032] Compare the pupil diameter of staff member i in the t-th frame facial image with the preset pupil diameter threshold of staff member i. If the pupil diameter of staff member i in the t-th frame facial image is greater than or equal to the preset pupil diameter threshold of staff member i, set the pupil size feature value to 0; otherwise, set the pupil size feature value to 1;

[0033] The preset pupil diameter threshold of the staff member i is specifically:

[0034] First, the pupil diameter of worker i when he is working normally and not sleepy in the preset second time period is obtained, and the average value of all pupil diameters is obtained, and the average value of the pupil diameters is multiplied by 0.4 as the preset pupil diameter threshold of worker i;

[0035] Step 213: Use the blinking frequency of worker i to obtain the blinking frequency characteristic value:

[0036] Compare the blinking frequency of the worker i in the preset third time period with the preset blinking frequency threshold, if the blinking frequency of the worker i in the preset third time period is less than or equal to the preset blinking frequency threshold, assign the blinking frequency characteristic value to 1; otherwise, assign the blinking frequency characteristic value to 0;

[0037] The preset blink frequency threshold is obtained by: obtaining the blink frequencies of worker i in multiple time periods when he is working normally and not sleepy, and obtaining an average value of the multiple blink frequencies, and multiplying the average value of the blink frequencies by 0.5 as the preset blink frequency threshold;

[0038] Step 214: Use the eye closure time of worker i to obtain the eye closure time feature value:

[0039] Get the duration of each eye closure. If the duration of each eye closure within five minutes is greater than the preset duration threshold, assign the eye closure duration feature value to 1; otherwise, assign the eye closure duration feature value to 0.

[0040] The preset continuous closing time threshold is 0.3 seconds;

[0041] Step 215: Use the eye opening amplitude characteristic value, pupil size characteristic value, blink frequency characteristic value, and eye closing time characteristic value of staff member i to obtain the staff member's eye characteristic value, specifically:

[0042] If at least one of the eye opening amplitude characteristic value, pupil size characteristic value, pupil size characteristic value, and eye closing time characteristic value of worker i is 1, then the eye characteristic value of worker i is 1; otherwise, the eye characteristic value of worker i is 0;

[0043] Step 22: using the facial features of the staff member to obtain the eyebrow feature value of the staff member;

[0044] Step 23: Use the facial features of the staff to obtain the head feature values ​​of the staff;

[0045] Step 24: Using the eye feature values, eyebrow feature values ​​and head feature values ​​of the staff, obtain the sleepiness index value corresponding to the staff's facial features, specifically:

[0046] Facial features corresponding to the sleepiness index value = w1 × eye feature value + w2 × eyebrow feature value + w3 × head feature value

[0047] Among them, w1 is the eye feature value weight, w2 is the eyebrow feature value weight, w3 is the head feature value weight, w1>w3>w2.

[0048] Furthermore, the step 22 of obtaining the eyebrow feature value of the staff member by using the facial features of the staff member is specifically as follows:

[0049] First, the position coordinates of the eyebrow region of worker i in the t-th frame facial image are obtained, and one-third of the eyebrow regions with large absolute values ​​of horizontal coordinates are obtained as the eyebrow regions to be fitted, and the eyebrow regions to be fitted are fitted into eyebrow segments;

[0050] Then, the eyebrow segment position of worker i in the t-th frame facial image is compared with the preset eyebrow segment position. If the eyebrow segment position deviates downward from the preset eyebrow segment position by more than the preset eyebrow offset threshold, the eyebrow feature value is assigned a value of 1; otherwise, the eyebrow feature value is assigned a value of 0;

[0051] The preset eyebrow segment position is obtained in the following manner: taking the face as the plane direction and the nose tip as the coordinate origin, obtaining the eyebrow segment position of the staff member i at different times when he is working normally, and taking the average coordinate of the eyebrow segment position at all times as the preset eyebrow segment position.

[0052] Furthermore, the steps 2 and 3 of obtaining the head feature values ​​of the staff member by using the facial features of the staff member are specifically as follows:

[0053] Step 231: Get the head shaking feature value of worker i:

[0054] Obtain the horizontal coordinate movement distance of each head position change of worker i within the preset fourth time period. If the horizontal coordinate movement distance of the head position change at the current moment is greater than or equal to the horizontal coordinate movement distance threshold and the head position returns to the previous moment position more than n times within the preset fourth time period, assign the head shaking feature value of worker i to 1; otherwise, assign the head shaking feature value of worker i to 0;

[0055] Wherein, n is an integer greater than or equal to 2;

[0056] Step 232: Get the nodding feature value of staff member i:

[0057] Obtain the vertical coordinate movement distance of each head position change of worker i within the preset fifth time period. If the vertical coordinate movement distance of the head position change at the current moment is greater than or equal to the vertical coordinate movement distance threshold and the head position returns to the previous moment position more than n' times within the preset fifth time period, assign the head shaking feature value of worker i to 1; otherwise, assign the head shaking feature value of worker i to 0;

[0058] Wherein, n' is an integer greater than or equal to 2;

[0059] Step 233: Obtain the staff member's head shaking feature value and the staff member's nodding feature value. If at least one feature value is 1, the staff member's head feature value is set to 1; otherwise, the staff member's head feature value is set to 0.

[0060] Furthermore, the step 3 uses the physical characteristics and body video of the staff member to obtain the sleepiness index value corresponding to the physical characteristics of the staff member, specifically:

[0061] Step 31: Obtain the trunk inclination characteristic value of worker i:

[0062] Obtain the percentage a1 of the face area of ​​worker i to the head area in the t-th frame body image; if a1 is less than the preset percentage, it is considered that the current worker is in a backward leaning state or a forward leaning state; when the worker is in a backward leaning state or a forward leaning state, the trunk inclination characteristic value is assigned to 1; otherwise, the trunk inclination characteristic value is assigned to 0;

[0063] Step 32: Get the body shaking characteristic value of worker i:

[0064] Obtain the horizontal coordinate movement distance of the trunk coordinate position of worker i within the preset sixth time period. If the horizontal coordinate movement distance of the trunk position change at the current moment is greater than or equal to the preset trunk movement threshold and the trunk position returns to the previous moment position more than n* times within the preset sixth time period, the table assigns the body shaking characteristic value of worker i to 1; otherwise, the body shaking characteristic value of worker i is assigned to 0;

[0065] Wherein, n* is an integer greater than or equal to 2;

[0066] Step 3. Use the trunk inclination characteristic value and body shaking characteristic value of worker i to obtain the sleepiness state index value corresponding to the worker's physical characteristics:

[0067] Physical characteristics corresponding to sleepiness index = w1'×trunk tilt characteristic value+w2'×body shaking characteristic value

[0068] Among them, w1' is the weight of the trunk tilt feature, and w2' is the weight of the body sway feature.

[0069] Furthermore, the step 4 uses the physical characteristics and body video of the staff member to obtain the health status index value corresponding to the physical characteristics of the staff member, specifically:

[0070] Step 41: Obtain chest pain characteristic value:

[0071] Determine whether the overlap time between the hand position and the chest position of staff member i exceeds the preset first time threshold; if the overlap time exceeds the preset first time threshold, assign the chest pain feature value to 1; otherwise, assign the chest pain feature value to 0;

[0072] Step 42: Get the characteristic value of head pain:

[0073] Determine whether the overlap time between the hand position and the head position of the staff member i exceeds the preset second time threshold; if it exceeds the preset second time threshold, assign the head pain characteristic value to 1; otherwise, assign the head pain characteristic value to 0;

[0074] Step 43: Obtain lodging characteristic values:

[0075] Determine whether the distance between the torso position of worker i and the ground plane coordinates is less than a preset distance threshold. If the distance between the torso position of worker i and the ground plane coordinates is less than or equal to the preset distance threshold, set the lodging characteristic value to 1; otherwise, set the lodging characteristic value to 0;

[0076] Step 4. Use the chest pain characteristic value, head pain characteristic value and prostration characteristic value to obtain the health status index value corresponding to the staff member's physical characteristics:

[0077] The health status index value corresponding to the physical characteristics = w1*×chest pain characteristic value + w2*×head pain characteristic value + w3*×lodging characteristic value

[0078] Among them, w1* is the weight of chest pain feature, w2* is the weight of head pain feature, and w3* is the weight of prostration feature.

[0079] Furthermore, the step 5 of using the physiological index value of the staff member to obtain the health status index value corresponding to the physiological index value is specifically:

[0080] According to the physiological index values ​​of staff member i, existing instruments are used to determine whether staff member i has symptoms of palpitations, heart palpitations, bradycardia, infarction, arrest, ventricular fibrillation, arrhythmia, and angina pectoris. If at least one symptom occurs, the health status index value of staff member i is set to 1; otherwise, the health status index value of staff member i is set to 0.

[0081] Furthermore, the alarm module determines the current status of the staff member according to the staff member status index value and issues an alarm, specifically:

[0082] If the sleepiness index value corresponding to the staff member's facial features is greater than or equal to 1 and the sleepiness index value corresponding to the staff member's physical features is greater than or equal to 1, it means that the current staff member is in a sleepy state and a sleepiness alarm is issued;

[0083] If the staff's physiological indicator value is greater than or equal to 1 or the health status index value corresponding to the staff's physical characteristics is greater than or equal to 1, it means that the staff is at health risk and a health risk alert is issued.

[0084] The beneficial effects of the present invention are:

[0085] The present invention combines multimodal monitoring technology to identify different characteristics of staff members, and realizes all-round and multi-dimensional monitoring of the status of power industry staff members through eye characteristics, body characteristics and physiological index characteristics, and can accurately capture and identify changes in the status of personnel, thereby significantly improving the accuracy of the working status of staff members. The system of the present invention can automatically identify changes in the status of staff members, improve the intelligence of the status monitoring of power industry staff members, and at the same time improve the monitoring efficiency of the status of staff members; the present invention sets a comparison threshold value according to the characteristic values ​​of actual staff members when they work normally, taking into account the differences between different staff members, and improving the accuracy of staff status monitoring, thereby reducing the risk of accidents caused by staff errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is the system architecture diagram of this product;

[0087] Figure 2 It is the work flow chart of the present invention. DETAILED DESCRIPTION

[0088] Specific implementation method 1: Figure 1-2 As shown, in this embodiment, an intelligent monitoring system for the status of power industry workers based on multimodal monitoring includes: a camera module, a contact sensor module, a status analysis module, and an alarm module;

[0089] The camera module is used to obtain facial video and body video of the staff member, and send the facial video and body video to the status analysis module;

[0090] The contact sensor module is used to obtain the physiological index values ​​of the staff and send the physiological index values ​​of the staff to the state analysis module;

[0091] The physiological index values ​​include: heart rate information, blood pressure information, and blood oxygen saturation;

[0092] The state analysis module is used to obtain the staff state index value based on the staff's facial video, body video and physiological index value, and send the staff state index value to the alarm module;

[0093] The alarm module determines the current status of the staff member according to the staff member status index value and issues an alarm.

[0094] Specific implementation method 2: The state analysis module is used to obtain the staff state index value based on the staff's facial video, body video and physiological index value, and send the staff state index value to the alarm module, specifically:

[0095] Step 1: Extract each frame of the staff member's facial video and body video, pre-process each frame, and extract the staff member's facial features and body features, specifically:

[0096] First, extract each frame of the staff member's facial video and body video, and convert each facial image and body image into a grayscale image;

[0097] Then, facial features are obtained using facial grayscale images and facial videos;

[0098] Facial features include: distance between upper eyelid and lower eyelid, pupil diameter, eyebrow position, head position, eye closure time, blinking frequency per unit time;

[0099] Finally, the body features are obtained using the body grayscale image;

[0100] Physical characteristics include: torso position, hand position, head position;

[0101] Step 2: Use the facial features and facial video of the staff member to obtain the sleepiness index value corresponding to the facial features of the staff member, specifically:

[0102] Step 21. Use the facial features of the staff to obtain the staff's eye feature values. The "eye feature value" is related to the "eye opening amplitude feature value", "pupil feature value", "blinking frequency feature value" and "eye closing feature value", specifically:

[0103] Step 2: Using the distance between the upper eyelid and the lower eyelid of worker i in the t-th frame facial image, obtain the eye opening amplitude feature value of worker i in the t-th frame facial image:

[0104] A1. Obtain the vertical distance H between the upper eyelid edge and the lower eyelid edge of worker i in the t-th frame facial image. i-t ;

[0105] A2. Get the preset eye-opening distance of worker i:

[0106] Obtain the vertical distance between the upper eyelid edge and the lower eyelid edge when the worker i works normally and is not sleepy during the preset first time period, and obtain the average value of all vertical distances between the upper eyelid edge and the lower eyelid edge as the preset eye opening distance of the worker i;

[0107] "Normal" as mentioned in this document means that the worker is not sleepy, not in danger or sick, and the work he / she handles will not make him / her nervous, angry, anxious, etc.

[0108] A3. H i-t Compared with the preset eye opening distance of 0.4 times, if H i-t If the preset eye-opening distance is greater than 0.4 times, it means that the staff member is not drowsy in the facial image of the tth frame, and the eye-opening amplitude feature value is set to 0; if H i-t If the preset eye-opening distance is less than or equal to 0.4 times, it means that the staff member is drowsy in the facial image of the tth frame, and the eye-opening amplitude feature value is set to 1;

[0109] Step 212: Use the pupil diameter of worker i in the t-th frame facial image to obtain the pupil size feature value:

[0110] The pupil diameter value of worker i in the t-th frame facial image is compared with the preset pupil diameter threshold of worker i. If the pupil diameter of worker i in the t-th frame facial image is greater than or equal to the preset pupil diameter threshold of worker i, it means that the worker is not drowsy, and the pupil size feature value is set to 0; otherwise, the pupil size feature value is set to 1;

[0111] The preset pupil diameter threshold of the staff member i is specifically:

[0112] First, the pupil diameter of worker i when he is working normally and not sleepy in the preset second time period is obtained, and the average value of all pupil diameters is obtained, and the average value of the pupil diameters is multiplied by 0.4 as the preset pupil diameter threshold of worker i;

[0113] Step 213: Obtain the blink frequency characteristic value using the blink frequency of worker i in the preset third time period:

[0114] The blinking frequency of the worker i in the preset third time period is compared with the preset blinking frequency threshold. If the blinking frequency of the worker i in the preset third time period is less than or equal to the preset blinking frequency threshold, it means that the worker is in a sleepy state in the preset third time period, and the blinking frequency characteristic value is assigned to 1; otherwise, the blinking frequency characteristic value is assigned to 0;

[0115] The preset blink frequency threshold is obtained by: obtaining the blink frequency of worker i in multiple time periods when he is working normally and not sleepy, and obtaining the average value of the multiple blink frequencies, and multiplying the average value of the blink frequency by 0.5 as the preset blink frequency threshold;

[0116] Step 214: Use the eye closure time of worker i to obtain the eye closure time feature value:

[0117] Get the duration of each eye closure. If the duration of each eye closure within 5 minutes is greater than the preset duration threshold, it means that worker i is drowsy, and the eye closure duration feature value is assigned to 1; otherwise, the eye closure duration feature value is assigned to 0;

[0118] The preset continuous closing time threshold is 0.3 seconds;

[0119] Step 215: Obtain the eigenvalue results obtained in steps 211 to 214. If at least one eigenvalue is 1, the eye eigenvalue of staff member i is 1; otherwise, it is 0.

[0120] Step 22: Use the facial features of the staff to obtain the eyebrow feature values ​​of the staff:

[0121] First, the position coordinates of the eyebrow region of worker i in the t-th frame facial image are obtained, and one-third of the eyebrow regions with large absolute values ​​of horizontal coordinates are obtained as the eyebrow regions to be fitted, and the eyebrow regions to be fitted are fitted into eyebrow segments; the eyebrow segment form is y=ax+b;

[0122] Then, the eyebrow segment position of worker i in the t-th frame facial image is compared with the preset eyebrow segment position. If the eyebrow segment position deviates downward from the preset eyebrow segment position by more than the preset eyebrow offset threshold, the eyebrow feature value is assigned a value of 1; otherwise, the eyebrow feature value is assigned a value of 0;

[0123] The preset eyebrow segment position is obtained by: taking the face as the plane direction and the nose tip as the coordinate origin, obtaining the eyebrow segment position of the staff member i at different times when he / she is working normally, and taking the average coordinate of the eyebrow segment position at all times as the preset eyebrow segment position;

[0124] The preset eyebrow offset threshold is 3 mm.

[0125] Step 23: Use the facial features of the staff to obtain the head feature values ​​of the staff:

[0126] Step 231: Get the head shaking feature value of worker i:

[0127] Obtain the horizontal coordinate movement distance of each head position change of worker i within the preset fourth time period. If the horizontal coordinate movement distance of the head position change at the current moment is greater than or equal to the horizontal coordinate movement distance threshold of 3 cm and the head position returns to this position more than twice within the preset fourth time period of 10 seconds, it means that worker i is drowsy, and the head shaking characteristic value of worker i is assigned to 1; otherwise, the head shaking characteristic value of worker i is assigned to 0;

[0128] Step 232: Get the staff nod feature value:

[0129] Obtain the vertical coordinate movement distance of each head position change of worker i within the preset fifth time period. If the vertical coordinate movement distance of the head position change at the current moment is greater than or equal to the vertical coordinate movement distance threshold of 3 cm and the head position returns to this position twice within 10 seconds, it means that worker i is drowsy, and the worker nodding feature value is assigned to 1; otherwise, the worker nodding feature value is assigned to 0;

[0130] Step 233: Obtain the staff member's head shaking feature value and the staff member's nodding feature value. If at least one feature value is 1, the staff member's head feature value is set to 1; otherwise, the staff member's head feature value is set to 0;

[0131] Step 24: Use the eye feature values, eyebrow feature values ​​and head feature values ​​of the staff member to obtain the sleepiness state index value corresponding to the staff member's facial features:

[0132] Facial features corresponding to the sleepiness index value = w1 × eye feature value + w2 × eyebrow feature value + w3 × head feature value

[0133] Among them, w1 is the weight of the eye feature value, w2 is the weight of the eyebrow feature value, w3 is the weight of the head feature value, w1>w3>w2;

[0134] Step 3: Use the staff's physical characteristics and body video to obtain the sleepiness index value corresponding to the staff's physical characteristics:

[0135] Step 31: Obtain the trunk inclination characteristic value of worker i:

[0136] Obtain the percentage a1 of the face area of ​​worker i to the head area in the body image of the t-th frame; if a1 is less than the preset percentage of 30%, it is considered that the current worker is in a backward leaning state or a forward leaning state; when the worker is in a backward leaning state or a forward leaning state, the trunk inclination characteristic value is assigned to 1; otherwise, the trunk inclination characteristic value is assigned to 0;

[0137] Step 32: Get the body shaking characteristic value of worker i:

[0138] Obtain the horizontal coordinate movement distance of the trunk coordinate position of worker i within the preset sixth time period. If the horizontal coordinate movement distance of the trunk position change at the current moment is greater than or equal to the preset trunk movement threshold of 5 cm and the trunk is placed back to this position more than twice within 10 seconds, it means that worker i is drowsy, and the body shaking characteristic value of worker i is assigned to 1; otherwise, the body shaking characteristic value of worker i is assigned to 0;

[0139] Step 3. Use the trunk inclination characteristic value and body shaking characteristic value of worker i to obtain the sleepiness state index value corresponding to the worker's physical characteristics:

[0140] Physical characteristics corresponding to sleepiness index = w1'×trunk tilt characteristic value+w2'×body shaking characteristic value

[0141] Among them, w1' is the weight of the trunk tilt feature, and w2' is the weight of the body sway feature;

[0142] Step 4: Using the physical characteristics and body video of the staff member, obtain the health status index value corresponding to the physical characteristics of the staff member;

[0143] Step 41: Obtain chest pain characteristic value:

[0144] It is determined that the overlap time between the hand position and the chest position of the staff member i exceeds the preset first time threshold of 5s; if it exceeds the preset first time threshold, it means that the staff member i has chest pain symptoms, and the chest pain feature value is assigned to 1; otherwise, the chest pain feature value is assigned to 0;

[0145] Step 42: Get the characteristic value of head pain:

[0146] Determine whether the overlap time between the hand position and the head position of the staff member i exceeds the preset second time threshold of 5s; if it exceeds the preset second time threshold, it means that the staff member i has a headache symptom, and the headache characteristic value is assigned to 1; otherwise, the headache characteristic value is assigned to 0;

[0147] Step 43: Obtain lodging characteristic values:

[0148] Determine whether the distance between the torso position of worker i and the ground plane coordinates is less than a preset distance threshold. If the distance between the torso position of worker i and the ground plane coordinates is less than or equal to the preset distance threshold, it means that worker i is in a prone state, and the prone characteristic value is set to 1; otherwise, the prone characteristic value is set to 0;

[0149] Step 4. Use the chest pain characteristic value, head pain characteristic value and prostration characteristic value to obtain the health status index value corresponding to the staff member's physical characteristics:

[0150] The health status index value corresponding to the physical characteristics = w1*×chest pain characteristic value + w2*×head pain characteristic value + w3*×losing characteristic value, where w1* is the chest pain characteristic weight, w2* is the head pain characteristic weight, and w3* is the losing characteristic weight;

[0151] Step 5: Use the physiological index values ​​of the staff to obtain the health status index values ​​corresponding to the physiological index values:

[0152] Use existing instruments to determine whether staff member i has symptoms of palpitations, panic, bradycardia, mild angina, infarction, arrest, ventricular fibrillation, severe arrhythmia, or severe angina. If any one or more symptoms occur, the staff member's health status index value is set to 1.

[0153] Step 6: Send the sleepiness index value corresponding to the facial features of the staff member, the sleepiness index value corresponding to the physical features of the staff member, the health index value corresponding to the physical features of the staff member, and the health index value obtained from the physiological index values ​​of the staff member to the alarm module;

[0154] Specific implementation method three: The alarm module determines the current status of the staff member according to the staff member status index value and issues an alarm, specifically:

[0155] If the sleepiness index value corresponding to the staff member's facial features is greater than or equal to 1 and the sleepiness index value corresponding to the staff member's physical features is greater than or equal to 1, it means that the current staff member is in a sleepy state and a sleepiness alarm is issued;

[0156] If the staff's physiological indicator value is greater than or equal to 1 or the health status index value corresponding to the staff's physical characteristics is greater than or equal to 1, it means that the staff is at health risk and a health risk alert is issued.

[0157] When working, all modules of the present invention are in synchronous operation, and all operation steps are executed cyclically.

Claims

1. An intelligent monitoring system for power industry workers based on multi-modal monitoring, characterized in that The system includes: a camera module, a contact sensor module, a state analysis module, and an alarm module; The camera module is used to obtain facial video and body video of the staff member, and send the facial video and body video to the status analysis module; The contact sensor module is used to obtain the physiological index values ​​of the staff and send the physiological index values ​​of the staff to the state analysis module; The physiological index values ​​include: heart rate information, blood pressure information, and blood oxygen saturation; The state analysis module is used to obtain the staff state index value based on the staff's facial video, body video and physiological index value, and send the staff state index value to the alarm module; The alarm module determines the current status of the staff member according to the staff member status index value and issues an alarm.

2. According to claim 1, the intelligent monitoring system for power industry workers based on multimodal monitoring is characterized by: The state analysis module is used to obtain the staff state index value based on the staff's facial video, body video and physiological index value, and send the staff state index value to the alarm module, specifically: Step 1: extract each frame of the staff member's facial video and body video, pre-process each frame, and extract the staff member's facial features and body features; Step 2: Using the facial features and facial video of the staff member, obtain the sleepiness index value corresponding to the facial features of the staff member; Step 3: Using the physical characteristics and body video of the staff, obtain the sleepiness index value corresponding to the physical characteristics of the staff; Step 4: Using the physical characteristics and body video of the staff member, obtain the health status index value corresponding to the physical characteristics of the staff member; Step 5: Using the physiological index values ​​of the staff, obtain the health status index values ​​corresponding to the physiological index values; Step 6: Send the sleepiness index value corresponding to the facial features of the staff member, the sleepiness index value corresponding to the physical features of the staff member, the health status index value corresponding to the physical features of the staff member, and the health status index value corresponding to the physiological index value to the alarm module.

3. According to claim 2, the intelligent monitoring system for power industry workers based on multimodal monitoring is characterized by: The step 1 extracts each frame of the staff member's facial video and body video, preprocesses each frame, and extracts the staff member's facial features and body features, specifically: First, extract each frame of the staff member's facial video and body video, and convert each facial image and body image into a grayscale image; Then, facial grayscale images and facial videos are used to obtain the facial features of the staff; Facial features include: distance between upper eyelid and lower eyelid, pupil diameter, eyebrow position, head position, eye closure time, blinking frequency per unit time; Finally, the body features are obtained using the body grayscale image; The body features include: torso position, hand position, and head position.

4. According to claim 3, the intelligent monitoring system for power industry workers based on multi-modal monitoring is characterized by: The step 2 of obtaining the sleepiness index value corresponding to the facial features of the staff member by using the facial features and the facial video of the staff member is specifically: Step 21: Use the facial features of the staff to obtain the eye feature values ​​of the staff, specifically: Step 2: Using the distance between the upper eyelid and the lower eyelid of worker i in the t-th frame facial image, obtain the eye opening amplitude feature value of worker i in the t-th frame facial image, specifically: A1. Obtain the vertical distance H between the upper eyelid edge and the lower eyelid edge of worker i in the t-th frame facial image. i-t ; A2. Get the preset eye-opening distance of worker i: Obtain the vertical distance between the upper eyelid edge and the lower eyelid edge when the worker i works normally and is not sleepy during the preset first time period, and obtain the average value of all vertical distances between the upper eyelid edge and the lower eyelid edge as the preset eye opening distance of the worker i; A3. H i-t Compared with the preset eye opening distance of 0.4 times, if H i-t If the eye opening distance is greater than 0.4 times the preset eye opening distance, the eye opening amplitude characteristic value is set to 0; if H i-t If the eye opening distance is less than or equal to 0.4 times the preset eye opening distance, the eye opening amplitude characteristic value is set to 1; Step 2: Use the pupil diameter of worker i in the t-th frame facial image to obtain the pupil size feature value: Compare the pupil diameter of staff member i in the t-th frame facial image with the preset pupil diameter threshold of staff member i. If the pupil diameter of staff member i in the t-th frame facial image is greater than or equal to the preset pupil diameter threshold of staff member i, set the pupil size feature value to 0; otherwise, set the pupil size feature value to 1; The preset pupil diameter threshold of the staff member i is specifically: First, the pupil diameter of worker i when he is working normally and not sleepy in the preset second time period is obtained, and the average value of all pupil diameters is obtained, and the average value of the pupil diameters is multiplied by 0.4 as the preset pupil diameter threshold of worker i; Step 2-3: Use the blinking frequency of worker i to obtain the blinking frequency characteristic value: Compare the blinking frequency of the worker i in the preset third time period with the preset blinking frequency threshold, if the blinking frequency of the worker i in the preset third time period is less than or equal to the preset blinking frequency threshold, assign the blinking frequency characteristic value to 1; otherwise, assign the blinking frequency characteristic value to 0; The preset blink frequency threshold is obtained by: obtaining the blink frequencies of worker i in multiple time periods when he is working normally and not sleepy, and obtaining an average value of the multiple blink frequencies, and multiplying the average value of the blink frequencies by 0.5 as the preset blink frequency threshold; Step 214: Use the eye closure time of worker i to obtain the eye closure time feature value: Get the duration of each eye closure. If the duration of each eye closure within five minutes is greater than the preset duration threshold, assign the eye closure duration feature value to 1; otherwise, assign the eye closure duration feature value to 0. The preset continuous closing time threshold is 0.3 seconds; Step 215: Use the eye opening amplitude characteristic value, pupil size characteristic value, blink frequency characteristic value, and eye closing time characteristic value of staff member i to obtain the staff member's eye characteristic value, specifically: If at least one of the eye opening amplitude characteristic value, pupil size characteristic value, pupil size characteristic value, and eye closing time characteristic value of worker i is 1, then the eye characteristic value of worker i is 1; otherwise, the eye characteristic value of worker i is 0; Step 22: using the facial features of the staff member to obtain the eyebrow feature value of the staff member; Step 23: Use the facial features of the staff to obtain the head feature values ​​of the staff; Step 24: Using the eye feature values, eyebrow feature values ​​and head feature values ​​of the staff, obtain the sleepiness index value corresponding to the facial features of the staff, specifically: Facial features corresponding to the sleepiness index value = w1 × eye feature value + w2 × eyebrow feature value + w3 × head feature value Among them, w1 is the weight of the eye feature value, w2 is the weight of the eyebrow feature value, w3 is the weight of the head feature value, w1>w3>w2.

5. According to claim 4, the intelligent monitoring system for power industry workers based on multi-modal monitoring is characterized by: The step 22 of obtaining the eyebrow feature value of the staff member by using the facial features of the staff member is specifically as follows: First, the position coordinates of the eyebrow region of worker i in the t-th frame facial image are obtained, and one-third of the eyebrow regions with large absolute values ​​of horizontal coordinates are obtained as the eyebrow regions to be fitted, and the eyebrow regions to be fitted are fitted into eyebrow segments; Then, the eyebrow segment position of worker i in the t-th frame facial image is compared with the preset eyebrow segment position. If the eyebrow segment position deviates downward from the preset eyebrow segment position by more than the preset eyebrow offset threshold, the eyebrow feature value is assigned a value of 1; otherwise, the eyebrow feature value is assigned a value of 0; The preset eyebrow segment position is obtained in the following manner: taking the face as the plane direction and the nose tip as the coordinate origin, obtaining the eyebrow segment position of the staff member i at different times when he is working normally, and taking the average coordinate of the eyebrow segment position at all times as the preset eyebrow segment position.

6. The intelligent monitoring system for power industry workers' status based on multi-modal monitoring according to claim 5 is characterized by: The steps 2 and 3 of obtaining the head feature values ​​of the staff member by using the facial features of the staff member are specifically as follows: Step 231: Get the head shaking feature value of worker i: Obtain the horizontal coordinate movement distance of each head position change of worker i within the preset fourth time period. If the horizontal coordinate movement distance of the head position change at the current moment is greater than or equal to the horizontal coordinate movement distance threshold and the head position returns to the previous moment position more than n times within the preset fourth time period, assign the head shaking feature value of worker i to 1; otherwise, assign the head shaking feature value of worker i to 0; Wherein, n is an integer greater than or equal to 2; Step 232: Get the nodding feature value of staff member i: Obtain the vertical coordinate movement distance of each head position change of worker i within the preset fifth time period. If the vertical coordinate movement distance of the head position change at the current moment is greater than or equal to the vertical coordinate movement distance threshold and the head position returns to the previous moment position more than n' times within the preset fifth time period, assign the head shaking feature value of worker i to 1; otherwise, assign the head shaking feature value of worker i to 0; Wherein, n' is an integer greater than or equal to 2; Step 233: Obtain the staff member's head shaking feature value and the staff member's nodding feature value. If at least one feature value is 1, the staff member's head feature value is set to 1; otherwise, the staff member's head feature value is set to 0.

7. The intelligent monitoring system for power industry workers' status based on multi-modal monitoring according to claim 6 is characterized by: The step 3 uses the physical characteristics and body video of the staff member to obtain the sleepiness index value corresponding to the physical characteristics of the staff member, specifically: Step 31: Obtain the trunk inclination characteristic value of worker i: Obtain the percentage a1 of the face area of ​​worker i to the head area in the t-th frame body image; if a1 is less than the preset percentage, it is considered that the current worker is in a backward leaning state or a forward leaning state; when the worker is in a backward leaning state or a forward leaning state, the trunk inclination characteristic value is assigned to 1; otherwise, the trunk inclination characteristic value is assigned to 0; Step 32: Get the body shaking characteristic value of worker i: Obtain the horizontal coordinate movement distance of the trunk coordinate position of worker i within the preset sixth time period. If the horizontal coordinate movement distance of the trunk position change at the current moment is greater than or equal to the preset trunk movement threshold and the trunk position returns to the previous moment position more than n* times within the preset sixth time period, the table assigns the body shaking characteristic value of worker i to 1; otherwise, the body shaking characteristic value of worker i is assigned to 0; Wherein, n* is an integer greater than or equal to 2; Step 3. Use the trunk inclination characteristic value and body shaking characteristic value of worker i to obtain the sleepiness state index value corresponding to the worker's physical characteristics: Physical characteristics corresponding to sleepiness index = w1'×trunk tilt characteristic value+w2'×body shaking characteristic value Among them, w1' is the weight of the trunk tilt feature, and w2' is the weight of the body sway feature.

8. The intelligent monitoring system for power industry workers' status based on multi-modal monitoring according to claim 7 is characterized by: The step 4 uses the physical characteristics and body video of the staff member to obtain the health status index value corresponding to the physical characteristics of the staff member, specifically: Step 41: Obtain chest pain characteristic value: Determine whether the overlap time between the hand position and the chest position of staff member i exceeds the preset first time threshold; if the overlap time exceeds the preset first time threshold, assign the chest pain feature value to 1; otherwise, assign the chest pain feature value to 0; Step 42: Get the characteristic value of head pain: Determine whether the overlap time between the hand position and the head position of the staff member i exceeds a preset second time threshold; If the preset second time threshold is exceeded, the headache characteristic value is assigned a value of 1; otherwise, the headache characteristic value is assigned a value of 0; Step 43: Obtain lodging characteristic values: Determine whether the distance between the torso position of worker i and the ground plane coordinates is less than a preset distance threshold. If the distance between the torso position of worker i and the ground plane coordinates is less than or equal to the preset distance threshold, set the lodging characteristic value to 1; otherwise, set the lodging characteristic value to 0; Step 4. Use the chest pain characteristic value, head pain characteristic value and prostration characteristic value to obtain the health status index value corresponding to the staff member's physical characteristics: The health status index value corresponding to the physical characteristics = w1*×chest pain characteristic value + w2*×head pain characteristic value + w3*×lodging characteristic value Among them, w1* is the weight of chest pain feature, w2* is the weight of head pain feature, and w3* is the weight of prostration feature.

9. The intelligent monitoring system for power industry workers' status based on multi-modal monitoring according to claim 8 is characterized by: The step 5 of using the physiological index value of the staff member to obtain the health status index value corresponding to the physiological index value is specifically as follows: According to the physiological index values ​​of staff member i, existing instruments are used to determine whether staff member i has symptoms of palpitations, heart palpitations, bradycardia, infarction, arrest, ventricular fibrillation, arrhythmia, and angina pectoris. If at least one symptom occurs, the health status index value of staff member i is set to 1; otherwise, the health status index value of staff member i is set to 0.

10. The intelligent monitoring system for power industry workers' status based on multi-modal monitoring according to claim 9 is characterized by: The alarm module determines the current status of the staff member according to the staff member status index value and issues an alarm, specifically: If the sleepiness index value corresponding to the staff member's facial features is greater than or equal to 1 and the sleepiness index value corresponding to the staff member's physical features is greater than or equal to 1, it means that the current staff member is in a sleepy state and a sleepiness alarm is issued; If the staff's physiological indicator value is greater than or equal to 1 or the health status index value corresponding to the staff's physical characteristics is greater than or equal to 1, it means that the staff is at health risk and a health risk alert is issued.

Images