Firefighter fatigue detection method and device based on dynamic region attention mechanism

Abstract

The application provides a firefighter fatigue detection method and device based on a dynamic region attention mechanism. The method collects continuous frame videos through a fire helmet camera, and calculates the target region occlusion degree during preprocessing. Accordingly, the dynamic region attention mechanism is used in the backbone network, the target region is divided into discrete sections according to the occlusion degree, and the sampling multiple is adjusted in the up-sampling layer to enhance feature extraction and improve the positioning accuracy of the small target region of the face and joints. The method effectively improves the real-time performance and accuracy of fatigue detection in complex fire environments, provides accurate decision support for the command center, ensures the safety of firefighters, and improves the rescue efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of fatigue detection, and particularly relates to a firefighter fatigue detection method and device based on a dynamic region attention mechanism. BACKGROUND

[0002] With the increase in the number of fires and the increase in the complexity of rescue tasks, traditional fire helmets have been difficult to meet the needs of modern firefighting for rapid response and efficient rescue. In particular, physical fatigue of firefighters during frequent high-intensity training and rescue tasks is one of the important factors leading to injuries or even sacrifices in fire scenes. Therefore, real-time and accurate detection of the fatigue state of firefighters is of great significance for preventing accidents and improving rescue efficiency.

[0003] In the prior art, some intelligent fire helmets integrate heart rate monitors, muscle electrical sensors, electroencephalogram sensors, respiratory rate monitors, and body temperature sensors, and combine machine learning algorithms and deep learning algorithms to achieve detection of the fatigue state of firefighters. However, these methods generally have problems such as poor real-time performance and response speed, high equipment procurement and maintenance costs, low safety and flexibility of rescue operations, and insufficient detection accuracy. In particular, in complex and variable firefighting environments, such as the presence of light changes, flames, smoke, water vapor, and other interference factors, as well as long-term changes in firefighter behavior patterns and helmet obstruction, the recognition accuracy and real-time performance of the prior art face great challenges. In addition, although personnel fatigue detection technology has made some progress in fields such as logistics sorting, transportation, and coal mining, such as a picker fatigue recognition system based on YOLOv5 and a fatigue driving state detection method based on a lightweight YOLOv7-TMC network, these technologies still have problems such as insufficient adaptability to complex environments and limited detection accuracy when applied to firefighting scenarios.

[0004] In particular, in firefighting scenarios, due to the influence of factors such as smoke, fire scenes, water vapor, and helmet obstruction, the real-time performance and detection accuracy of existing methods in recognizing the fatigue state of firefighters still need to be improved. SUMMARY

[0005] The purpose of the present application is to overcome the defects in the prior art described above, and to provide a firefighter fatigue detection method and device based on a dynamic region attention mechanism.

[0006] The present application provides a firefighter fatigue detection method based on a dynamic region attention mechanism, comprising:

[0007] Continuous frame video data is obtained by a fire helmet camera, and the degree of obstruction based on a fire scene in a target region in each frame is calculated in a preprocessing stage;

[0008] According to the degree of occlusion based on the fire scene, a dynamic region attention mechanism is adopted in the backbone network, and the target region is divided into a segmentation mode with a discrete number of segments;

[0009] In the upsampling layer of the backbone network, the sampling multiple is adjusted to the same value according to the discrete number of segments;

[0010] Feature extraction is performed through the sampling multiple, and the positioning accuracy of the dynamic region attention mechanism for small target regions such as faces and joints is enhanced.

[0011] Optionally, the calculation of the degree of occlusion based on the fire scene in the target region of each frame in the preprocessing stage comprises:

[0012] The interference regions of smoke, water vapor and flame are located by the FireNet network;

[0013] The target region is adjusted based on the interference region, and is extracted frame by frame into an image;

[0014] The visible region proportion of the adjusted target region in each frame image is calculated, and the degree of occlusion based on the fire scene is divided according to the visible region proportion based on a preset occlusion level division standard.

[0015] Optionally, the dynamic region attention mechanism is adopted in the backbone network, and the target region is divided into a segmentation mode with a discrete number of segments, comprising:

[0016] When the degree of occlusion based on the fire scene is heavy occlusion, a ten-segment segmentation mode is selected;

[0017] When the degree of occlusion based on the fire scene is moderate occlusion, an eight-segment segmentation mode or a six-segment segmentation mode is selected;

[0018] When the degree of occlusion based on the fire scene is mild occlusion, a four-segment segmentation mode is selected.

[0019] Optionally, in the upsampling layer of the backbone network, the sampling multiple is adjusted to the same value according to the discrete number of segments, comprising:

[0020] According to the discrete number of segments, the current detection accuracy and detection speed requirement, a sampling multiple same as the discrete number of segments is selected;

[0021] Through the sampling multiple, the micro visible features in the target region are reconstructed based on the current detection accuracy and detection speed requirement.

[0022] Optionally, after the feature extraction through the sampling multiple and the enhancement of the positioning accuracy of the dynamic region attention mechanism for small target regions such as faces and joints, the method further comprises:

[0023] The positioning accuracy enhanced face and joint small target region features are input into a GRU model;

[0024] The time sequence change of the features is analyzed by the GRU model to determine the firefighter fatigue state;

[0025] According to the fatigue state and the time sequence change, a graded alarm signal is output to an indicator light module.

[0026] The application provides a firefighter fatigue detection device based on a dynamic region attention mechanism, comprising:

[0027] An occlusion module acquires continuous frame video data through a fire helmet camera, and calculates a fire scene-based occlusion degree of a target region in each frame in a preprocessing stage;

[0028] A division module divides the target region into a segmentation mode with a discrete number of segments in a backbone network according to the fire scene-based occlusion degree and using a dynamic region attention mechanism;

[0029] An adjustment module adjusts a sampling multiple to the same value according to the discrete number of segments in an up-sampling layer of the backbone network;

[0030] An execution module performs feature extraction through the sampling multiple to enhance the positioning accuracy of the dynamic region attention mechanism on the face and joint small target region.

[0031] Optionally, the occlusion module calculates the fire scene-based occlusion degree of the target region in each frame in the preprocessing stage, comprising:

[0032] A FireNet network is used to locate smoke, water vapor and flame interference regions;

[0033] The target region is adjusted based on the interference regions, and is extracted frame by frame into images;

[0034] The visible region proportion of the adjusted target region in each frame image is calculated, and the fire scene-based occlusion degree is divided according to the visible region proportion based on a preset occlusion level division standard.

[0035] Optionally, the adjustment module divides the target region into a segmentation mode with a discrete number of segments in a backbone network using a dynamic region attention mechanism, comprising:

[0036] When the fire scene-based occlusion degree is severe occlusion, a ten-segment segmentation mode is selected;

[0037] When the fire scene-based occlusion degree is moderate occlusion, an eight-segment segmentation mode or a six-segment segmentation mode is selected;

[0038] When the fire scene-based occlusion degree is slight occlusion, a four-section segmentation mode is selected.

[0039] Optionally, the adjusting module adjusts the sampling multiple to the same value according to the discrete section number in the up-sampling layer of the backbone network, comprising:

[0040] According to the discrete section number, the current detection accuracy and the detection speed requirement, a sampling multiple same as the discrete section number is selected;

[0041] Through the sampling multiple, the micro visible features in the target region are reconstructed based on the current detection accuracy and the detection speed requirement.

[0042] Optionally, the executing module executes feature extraction through the sampling multiple, and after enhancing the positioning accuracy of the face and joint small target region of the dynamic region attention mechanism, further comprising:

[0043] The features of the face and joint small target region after the positioning accuracy are enhanced are input into a GRU model;

[0044] The time sequence change of the features is analyzed through the GRU model to determine the fatigue state of the firefighter;

[0045] According to the fatigue state and the time sequence change, a graded alarm signal is output to an indicator light module.

[0046] The beneficial effects of the present application are:

[0047] The present application provides a firefighter fatigue detection method based on a dynamic region attention mechanism, comprising: acquiring continuous frame video data through a fire helmet camera, calculating the fire scene-based occlusion degree of a target region in each frame in a preprocessing stage; according to the fire scene-based occlusion degree, a dynamic region attention mechanism is used in a backbone network to divide the target region into a segmentation mode with a discrete section number; in the up-sampling layer of the backbone network, the sampling multiple is adjusted to the same value according to the discrete section number; feature extraction is performed through the sampling multiple to enhance the positioning accuracy of the dynamic region attention mechanism to the face and joint small target region. The present application enhances the positioning accuracy of the face and joint small target region of the firefighter through the dynamic region attention mechanism, effectively improves the real-time performance and accuracy of fatigue detection in complex fire environment, provides accurate decision basis for the command center, ensures the safety of the firefighter and improves the rescue efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of the firefighter fatigue detection method based on the dynamic region attention mechanism in the present application. DETAILED DESCRIPTION

[0049] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that various forms of the present disclosure are implemented without being limited by the embodiments set forth herein. Rather, the embodiments are provided so that the present disclosure can be more thoroughly understood, and so that the scope of the present disclosure can be conveyed to those skilled in the art.

[0050] Referring to Figure 1 The present application provides a safety-economy-green-based power system planning multi-dimensional evaluation method, which comprises the following steps:

[0051] S101: Obtain continuous frame video data through the fire helmet camera, and calculate the fire scene-based occlusion degree of the target region in each frame in the preprocessing stage.

[0052] Obtaining continuous frame video data through the fire helmet camera refers to using the camera installed on the fire helmet to collect continuous frame video data containing the firefighter's face, joints and environmental scene in real time when the firefighter performs the task.

[0053] The target region includes the facial features (such as eyes, lips, and nose tip) and joint parts (such as shoulders, elbows, wrists, knees, and ankles) of the firefighter.

[0054] In the preprocessing stage, the fire scene-based occlusion degree of the target region in each frame is calculated, which means that the continuous frame video data obtained is preprocessed to determine the degree of occlusion of the target region by smoke, water vapor, flame or rescue operation. Specifically, it includes: first, use the FireNet network to locate the smoke, water vapor and flame interference area in the video frame. The FireNet network accurately identifies the location of these interference areas by analyzing the texture features of smoke and water vapor and the color dynamic changes of flame. Then, adjust the target region based on the interference area, reduce the influence of the interference area on the target region through cropping or mask processing, and extract each frame of the adjusted video data into an image format. Finally, calculate the visible area ratio of the adjusted target region in each image. The visible area ratio refers to the proportion of the pixel area of the target region that is not covered by the interference area to the total pixel area.

[0055] Based on the preset occlusion level division standard, the visible area ratio is divided into fire scene-based occlusion degrees: when the visible area ratio is less than 30%, it is classified as severe occlusion; when the visible area ratio is 30%-70%, it is classified as moderate occlusion; and when the visible area ratio is greater than or equal to 70%, it is classified as light occlusion. For example, in the firefighter's face area, if the visible area ratio of the eye key point is less than 30%, it is determined that this area is severely occluded.

[0056] The preprocessing stage also includes normalizing the video data to unify the brightness range and performing EDSR super-resolution reconstruction to improve the clarity of low-quality frames, ensuring the accuracy of subsequent calculations of the degree of occlusion. The entire preprocessing stage runs on the NVIDIA Jetson AGX Orin hardware platform, using the OpenCV library to implement image conversion and scale calculation.

[0057] S102: According to the degree of occlusion based on the fire scene, a dynamic region attention mechanism is adopted in the backbone network to divide the target region into a segmentation mode with a discrete number of segments.

[0058] According to the degree of occlusion based on the fire scene, the calculated degree of occlusion (mild, moderate, or severe) is used as input. In the backbone network, a dynamic region attention mechanism is adopted, which replaces the original region attention module A2 in the backbone part of the YOLOv12-M model.

[0059] The target region is divided into a segmentation mode with a discrete number of segments, which means that the number of segments of the target region is adaptively selected according to the degree of occlusion: when the degree of occlusion is severe, a ten-segment segmentation mode is selected to divide the target region into ten equal blocks; when the degree of occlusion is moderate, an eight-segment segmentation mode or a six-segment segmentation mode is selected, with a preference for an eight-segment segmentation to enhance the focusing ability on moderate occlusion regions; when the degree of occlusion is mild, a four-segment segmentation mode is selected to divide the target region into four equal blocks.

[0060] The selection of the discrete number of segments is based on the fact that the more severe the degree of occlusion, the more segments there are, to enhance the focus on small visible regions. For example, in the firefighter's face region, if the nose tip is determined to be severely occluded, a ten-segment segmentation mode is used, and each block calculates the attention weight independently, thus capturing residual features more finely.

[0061] The dynamic region attention mechanism generates a segmentation mask through a convolution layer and adjusts the number of segments in real time according to the degree of occlusion, ensuring the detection ability of small target regions in complex scenes such as smoke and fire. The backbone network runs under the PyTorch2.4 framework and uses Python3.10.0 programming to implement the segmentation logic.

[0062] S103: In the upsampling layer of the backbone network, the sampling multiple is adjusted to the same value according to the discrete number of segments.

[0063] In the upsampling layer of the backbone network, the upsampling layer is responsible for feature amplification in the YOLOv12-M model.

[0064] According to the adjustment of the sampling multiple to the same number of discrete segments, the sampling multiple is set to be consistent with the number of discrete segments in S102: if the number of discrete segments is four, the sampling multiple is adjusted to four times; if the number of discrete segments is six or eight, the sampling multiple is adjusted to six times or eight times; if the number of discrete segments is ten, the sampling multiple is adjusted to ten times.

[0065] The adjustment process is comprehensively selected according to the current detection accuracy and detection speed requirement, for example, in a scene with high real-time requirement (such as rapid response to fire), a lower sampling multiple is preferentially selected to improve the speed; in a scene with high accuracy requirement (such as heavy occlusion), a higher sampling multiple is preferentially selected.

[0066] By the sampling multiple, the micro visible features in the target region are reconstructed based on the current detection accuracy and detection speed requirement, and the reconstruction means that the feature map is enlarged by using the interpolation algorithm (such as bilinear interpolation) of the upsampling layer to enhance the detail information of the micro region. For example, when the face key point region is divided into ten segments, the sampling multiple is adjusted to ten times, and the feature map is enlarged to retain the fine texture such as the edge of the iris of the eye. The adjustment process is monitored by the validation set in the training stage to ensure that the sampling multiple is matched with the number of discrete segments, and the parameters are fixed.

[0067] S104: Perform feature extraction by the sampling multiple to enhance the positioning accuracy of the dynamic region attention mechanism for the face and joint small target region.

[0068] Performing feature extraction by the sampling multiple means performing upsampling operation by using the adjusted sampling multiple to extract the feature map of the target region. Enhancing the positioning accuracy of the dynamic region attention mechanism for the face and joint small target region, the face region includes features such as eyes, lips, and nose tip, and the joint small target region includes key points such as shoulders, elbows, wrists, knees, and ankles.

[0069] The feature extraction process uses the convolution layer of the YOLOv12-M model to process the upsampled feature map, and outputs the coordinate position and bounding box information of the key point. The positioning accuracy is optimized by the a-SIOU positioning loss function, and the calculation formula is:

[0070] a-SIOU = angle penalty term + distance normalization term

[0071] The angle penalty term is used to correct the head posture angle error, and the distance normalization term is used to eliminate the influence of different scale targets.

[0072] The positioning accuracy after enhancement is controlled within 0.5 pixels, for example, in eye key point detection, the positioning error of the iris edge is less than 0.5 pixels. The feature extraction process combines the Mish activation function, and the smooth gradient characteristic of which retains fatigue-related features (such as pale or red blood vessels).

[0073] Furthermore, the features of the face and joint small target areas after the positioning accuracy is enhanced are input into the GRU model; the temporal changes of the features are analyzed through the GRU model to determine the fatigue state of the firefighter; and a graded alarm signal is output to the indicator light module according to the fatigue state and temporal changes.

[0074] After enhancing the positioning accuracy, the features of the facial and joint small target regions are input into the GRU model. These features include the eye aspect ratio (EAR), mouth aspect ratio (MAR), and joint angle sequence. The formulas for calculating EAR and MAR are as follows:

[0075]

[0076] Among them, the The left corner of the left eye, as described The point located on the upper left edge of the iris of the left eye. The point located on the upper right edge of the iris of the left eye. The point at the outer corner of the left eye and the point at the outer corner of the right eye, as described The point located on the lower right edge of the iris of the left eye. This is the lower left edge of the iris of the left eye.

[0077]

[0078] Among them, the The left corner of the mouth, the point described The point is the edge of the upper left lip. The center point of the upper lip, the The point on the upper right lip edge, the The right corner of the mouth, as described The point on the lower right lip edge, the The center point of the lower lip, the This is the point on the lower left lip edge.

[0079] The GRU model analyzes the temporal changes of the features. It uses an improved dynamic Huber loss function to analyze the temporal changes of features in consecutive frames, such as the cumulative duration of closing eyes, yawning, and slow walking (small changes in joint angles). The fatigue level of firefighters is determined by calculating the degree of fatigue using the PERCLOS-BDMT method.

[0080]

[0081] Among them, the For the feature cumulative duration, the The total duration (30 seconds, 1 minute or 3 minutes) is determined. When the PERCLOS-BDMT is less than 15%, it is judged to be a normal state, 15%-30% is mild fatigue, 30%-45% is moderate fatigue, and greater than 45% is severe fatigue. According to the fatigue state and timing change, a graded alarm signal is output to the indicator light module, and the graded alarm signal includes: a green indicator light indicating normal, a blue indicator light indicating mild fatigue, a yellow indicator light indicating moderate fatigue, and a red indicator light indicating severe fatigue; when the feature persists (such as closing the eyes for more than 50% of the total duration), the red and blue indicator lights alternate and flash to issue an emergency alarm. The alarm signal is output through the Raspberry Pi control GPIO interface, and the voice module broadcasts the specific firefighter number and fatigue degree in Mandarin (such as "001 moderate fatigue"). The GRU model runs on the Ubuntu20 system, and the PyTorch2.4 is used to realize the timing analysis logic.

[0082] Further, in the above continuous frame video data obtained by the fire helmet camera, it also includes:

[0083] A wide range of four continuous frame video data and related corresponding numbers of each firefighter are collected under the four states of mild fatigue, moderate fatigue, severe fatigue and normal of firefighters wearing fire helmets in the scenes of daily drills, daily on-duty and preparation state, and rapid response to fire scenes.

[0084] In addition, during the collection of data, the four videos also involve complex situations including smoke, fire, water, mild, moderate and severe occlusion caused by helmets or related rescue operations.

[0085] To enhance the generalization ability of the model and ensure the diversity and richness of the data set, so that the joint model can more easily learn and distinguish the corresponding features of firefighters in normal and the above three different fatigue levels during training, it is convenient to more accurately identify and distinguish the current firefighter in normal, mild fatigue, moderate fatigue and severe fatigue. For the continuous frame video data of the above three different fatigue levels, expand in turn through "time warping, random frame sampling, smoke, water and fire concentration gradual change, firefighter motion blur, video horizontal flip" and other operations, so that the total number of frame videos of the above three fatigue states is not only consistent with the number of frame videos in the normal state, but also makes the number of frame videos in the three fatigue states consistent respectively.

[0086] Further, in the above and the adjusted frame video data is extracted as an image format frame by frame, it also includes:

[0087] Using a computational vision annotation tool (CVAT), each single frame in the preprocessed image sequence was labeled with a rectangular bounding box, corresponding text labels, key point annotations, and timestamps based on corresponding facial features and key body nodes. Specific text labels included examples such as "Firefighter 1 - Normal face," "Firefighter 1 - Normal nose tip," "Firefighter 1 - Normal skin," "Firefighter 1 - Normal eyes," "Firefighter 1 - Pale face," "Firefighter 1 - Grayish face," "Firefighter 1 - Red nose tip," "Firefighter 1 - Dull face," "Firefighter 1 - Sallow skin," "Firefighter 1 - Dark circles," and "Firefighter 1 - Bloodshot eyes." Specific key point annotations included examples such as "Firefighter 1 - Left eye, left corner of eye point 1." The following points are listed: 1. Firefighter 1 - Upper left edge of left iris; 2. Firefighter 1 - Upper right edge of left iris; 3. Firefighter 1 - Right corner of left eye; 4. Firefighter 1 - Lower right edge of left iris; 5. Firefighter 1 - Lower left edge of left iris; 6. Firefighter 1 - Left corner of mouth; 1. Firefighter 1 - Left upper lip edge; 2. Firefighter 1 - Center of upper lip; 3. Firefighter 1 - Right upper lip edge; 4. Firefighter 1 - Right corner of mouth; 5. Firefighter 1 - Right lower lip edge; 6. Firefighter 1 - Center of lower lip; 7. Firefighter 1 - Left lower lip edge; 8. Firefighter 1 - Left shoulder joint; 9. Firefighter 1 - Center of left elbow; 10. Firefighter 1 - Left wrist; 11. Center of left knee; 12. Center of left ankle, etc.

[0088] Furthermore, to facilitate the subsequent explanation of the calculation methods for EAR and MAR, these corresponding markers for "eyes, lips, shoulders, elbows, wrists, knees, and ankles" are named sequentially as follows: , , , , , "", , , , , , , , “C”, “D”, “F”, “G”. The specific timestamp annotation is mainly based on the relative timestamps calculated in the preprocessing. For each image in the preprocessed image sequence, while performing the above-mentioned rectangle annotation, text box annotation and key point annotation, timestamp annotation is also performed.

[0089] Furthermore, the feature extraction process described above, which incorporates the Mish activation function, also includes:

[0090] Due to the RELU activation function in the CNN, when the input is negative, it is easy to cause the loss of relevant information such as "pale skin or face, sallow skin, red blood vessels in the eyes" related to fatigue features, so in order to retain these feature information, the Mish function will be used instead of the original RELU activation function in the CNN model, and the use of the Mish function gradient smoother characteristics will make the CNN model more easily capture the preliminary features of these relevant information and the preliminary position when the key points such as "eyes, lips, shoulders, elbows, wrists, knees and ankles" change continuously.

[0091] Further, in the above positioning accuracy optimization by a-SIOU positioning loss function, it also includes:

[0092] Because the α-SIOU function not only introduces an angle penalty term, but also introduces a distance normalization term to avoid the influence of different scale targets, and a super parameter a that can be continuously adjusted according to different angle losses and model tasks in different stages of training to balance positioning accuracy and convergence speed, etc., so that in this task of detecting whether the firefighter is tired, not only can the positioning accuracy of the YOLOv12-M model in the complex scene including the firefighter's helmet camera located at different distances, different head poses, different step angles and body postures caused by executing tasks, etc. It can also further improve the convergence speed of the YOLOv12-M model. Therefore, the a-SIOU function is also used instead of the CIOU function part of the original positioning loss function in the YOLOv12-M model.

[0093] Further, in the above GRU model using the improved dynamic Huber loss function, it also includes:

[0094] Because the dynamic or adaptive Huber loss function can automatically adjust and select the optimal threshold according to the real-time data characteristics of different tasks or the error distribution in the training process, to improve higher prediction accuracy or adapt to more scenes, so it has higher detection and classification accuracy, stronger adaptability to various tasks or scenes when the number of fatigue feature outliers is large, etc. Advantage, therefore, the dynamic or adaptive Huber loss function is used instead of the original mean square error function (MSE) in the model.

[0095] Further, in the above input features including eye aspect ratio EAR, mouth aspect ratio MAR, it also includes:

[0096] When performing key point labeling on the "eyes and lips" of each firefighter in the normal state, the overall EAR average of these firefighters when closing their eyes and opening their eyes in this state is also calculated (respectively abbreviated as: , , standard deviation (abbreviated as , ) and eye distance of each firefighter , and the overall MAR mean (abbreviated as: , ) and standard deviation (abbreviated as: , ) when normal opening, when not normal opening; wherein i represents the serial number of each firefighter. Select , , , respectively, as the initial threshold for judging the upper limit of each firefighter's closed eyes, the lower limit of open eyes, and the lower limit of full opening. Then search and optimize the three initial thresholds in the corresponding range of [ , ], [ , ], [ , ] respectively according to the EAR and MAR calculated above; and by monitoring the three threshold values searched in the validation set about the yawning classification accuracy of closed eyes, open eyes and full opening, the best EAR threshold value of each firefighter's closed eyes and open eyes and the MAR threshold value of yawning when full opening are selected.

[0097] Further, in the above classification output of the fatigue state and the time sequence change, the classification alarm signal is output to the indicator module, further comprising:

[0098] The alarm logic design method is as follows:

[0099] When the firefighter is judged to be in a fatigue state at least twice in a row, an alarm is triggered, and according to the specific fatigue degree of the last time, the corresponding color alarm indicator light is displayed according to the previous scheme, wherein the specific fatigue degrees involved in the two times can be different.

[0100] When the firefighter is judged to be in a normal state at least three times in a row, no alarm is triggered, and the alarm indicator light color corresponding to the fatigue degree is converted to green or a green indicator light is directly displayed. Among them, the measure of converting the alarm indicator light color is to adapt to the situation that the firefighter is adjusted from any one of the above three fatigue states to a normal state; directly displaying a green indicator light means that the firefighter has been in a normal state.

[0101] Further, in the above entire preprocessing stage running on the NVIDIA Jetson AGX Orin hardware platform, further comprising:

[0102] Before deployment, the following operations are performed on the deployed model:

[0103] Pruning refers to removing unimportant weights or connections in YOLOv12-M and GRU to reduce the complexity of the model and improve the inference speed of the model;

[0104] Quantization refers to converting floating-point weights and activation values in YOLOv12-M and GRU into low-precision integer form;

[0105] Model conversion mainly refers to converting the best performance model into a format executable in the actual application environment if the target deployment environment is not suitable for the framework used during training. For the PyTorch framework used this time, the ONNX tool is used to convert the evaluated joint model into an ONNX format suitable for execution under multiple deep learning frameworks or hardware platforms.

[0106] Installing a cross-compilation tool chain mainly compiles the model to be deployed and the deep learning framework into a binary file that can run in the actual application environment. If the target device is a server or a high-performance environment, cross-compilation is not required. However, the deep learning framework PyTorch and the corresponding dependencies in the present application need to be installed.

[0107] The application provides a firefighter fatigue detection device based on a dynamic region attention mechanism, comprising:

[0108] The occlusion module obtains continuous frame video data through a fire helmet camera, and calculates the occlusion degree of a target region based on a fire scene in each frame in a preprocessing stage.

[0109] The division module divides the target region into a segmentation mode with a discrete number of segments in a backbone network using a dynamic region attention mechanism according to the occlusion degree based on the fire scene.

[0110] The adjustment module adjusts the sampling multiple to the same value according to the discrete number of segments in the up-sampling layer of the backbone network.

[0111] The execution module performs feature extraction through the sampling multiple to enhance the positioning accuracy of the dynamic region attention mechanism for small target regions such as faces and joints.

[0112] Optionally, the occlusion module calculates the occlusion degree of the target region based on the fire scene in each frame in the preprocessing stage, comprising:

[0113] Locate the smoke, water vapor and flame interference area through the FireNet network;

[0114] Adjust the target region based on the interference area, and extract it frame by frame into an image;

[0115] calculate a visible region proportion of the adjusted target region in each frame of image, and divide the fire-fighting scene-based occlusion degree according to the visible region proportion based on a preset occlusion level division standard.

[0116] Optionally, the adjusting module adopts a dynamic region attention mechanism in the backbone network, divides the target region into a segmentation mode with a discrete segment number, including:

[0117] when the fire-fighting scene-based occlusion degree is severe occlusion, a ten-segment segmentation mode is selected;

[0118] when the fire-fighting scene-based occlusion degree is moderate occlusion, an eight-segment segmentation mode or a six-segment segmentation mode is selected;

[0119] when the fire-fighting scene-based occlusion degree is slight occlusion, a four-segment segmentation mode is selected.

[0120] Optionally, the adjusting module adjusts a sampling multiple to a same value according to the discrete segment number in an up-sampling layer of the backbone network, including:

[0121] selecting a sampling multiple same as the discrete segment number according to the discrete segment number, a current detection accuracy and a detection speed requirement;

[0122] reconstructing a micro- visible feature in the target region based on the current detection accuracy and the detection speed requirement through the sampling multiple.

[0123] Optionally, the executing module executes feature extraction through the sampling multiple, and after enhancing the positioning accuracy of the dynamic region attention mechanism on the small target region of the face and the joint, further includes:

[0124] inputting the feature of the small target region of the face and the joint after the positioning accuracy is enhanced into a GRU model;

[0125] analyzing a time sequence change of the feature through the GRU model to determine a firefighter fatigue state;

[0126] outputting a graded alarm signal to an indicator light module according to the fatigue state and the time sequence change.

[0127] The above description of the embodiments is to facilitate the general technical personnel in the art to understand and apply the present application. The skilled in the art can obviously make various modifications to the above embodiments, and apply the general principles described herein to other embodiments without having to have creative labor. Therefore, the present application is not limited to the above embodiments, and the improvements and modifications of the present application made by the skilled in the art according to the disclosure of the present application should be within the protection scope of the present application.

Claims

1. A method for detecting firefighter fatigue based on a dynamic region attention mechanism, characterized in that, include: The fire helmet camera acquires continuous frame video data, and the degree of occlusion of the target area in each frame based on the fire scene is calculated in the preprocessing stage. Based on the degree of occlusion in the fire-fighting scenario, a dynamic region attention mechanism is used in the backbone network to divide the target region into segments with discrete numbers of segments, including: when the degree of occlusion in the fire-fighting scenario is severe occlusion, a ten-segment segmentation method is selected; when the degree of occlusion in the fire-fighting scenario is moderate occlusion, an eight-segment segmentation method or a six-segment segmentation method is selected; when the degree of occlusion in the fire-fighting scenario is mild occlusion, a four-segment segmentation method is selected. In the upsampling layer of the backbone network, the sampling factor is adjusted to the same value according to the number of discrete segments; Feature extraction is performed by performing upsampling operation using the sampling multiple to enhance the positioning accuracy of the dynamic region attention mechanism for small target areas such as the face and joints; the features of the small target areas of the face and joints after the positioning accuracy is enhanced are input into the GRU model; the temporal changes of the features are analyzed by the GRU model to determine the fatigue state of the firefighter; and a graded alarm signal is output to the indicator light module according to the fatigue state and temporal changes.

2. The method for detecting firefighter fatigue based on a dynamic region attention mechanism according to claim 1, characterized in that, The calculation of the occlusion level of the target region in each frame based on the fire scene during the preprocessing stage includes: The FireNet network is used to locate areas of interference from smoke, water vapor, and flames. The FireNet network identifies the location of the interference areas by analyzing the texture features of smoke and water vapor and the dynamic color changes of flames. The target region is adjusted based on the interference region, and extracted as an image frame by frame; Calculate the proportion of the visible area of ​​the adjusted target region in each frame of the image, and classify the degree of occlusion based on the fire scene according to the proportion of the visible area based on the preset occlusion level classification standard.

3. The method for detecting firefighter fatigue based on a dynamic region attention mechanism according to claim 1, characterized in that, In the upsampling layer of the backbone network, adjusting the sampling factor to the same value according to the number of discrete segments includes: Based on the number of discrete segments, the current detection accuracy, and the required detection speed, select a sampling multiple that is the same as the number of discrete segments; By using the sampling factor, the minute visible features in the target area are reconstructed based on the current detection accuracy and detection speed requirements.

4. A firefighter fatigue detection device based on a dynamic region attention mechanism, characterized in that, include: The occlusion module acquires continuous frame video data through the fire helmet camera and calculates the degree of occlusion of the target area in each frame based on the fire scene during the preprocessing stage. The segmentation module, based on the degree of occlusion in the fire-fighting scenario, employs a dynamic region attention mechanism in the backbone network to divide the target region into segmentation methods with discrete segments, including: when the degree of occlusion in the fire-fighting scenario is severe occlusion, a ten-segment segmentation method is selected; when the degree of occlusion in the fire-fighting scenario is moderate occlusion, an eight-segment segmentation method or a six-segment segmentation method is selected; when the degree of occlusion in the fire-fighting scenario is mild occlusion, a four-segment segmentation method is selected. The adjustment module adjusts the sampling factor to the same value according to the number of discrete segments in the upsampling layer of the backbone network; The execution module performs upsampling operations to extract features by increasing the sampling multiple, thereby enhancing the positioning accuracy of the dynamic region attention mechanism for small target areas such as the face and joints. After the positioning accuracy is enhanced, the features of the small target areas of the face and joints are input into the GRU model. The GRU model analyzes the temporal changes of the features to determine the fatigue state of the firefighter. Based on the fatigue state and temporal changes, a graded alarm signal is output to the indicator light module.

5. A firefighter fatigue detection device based on a dynamic region attention mechanism according to claim 4, characterized in that, The occlusion module calculates the degree of occlusion of the target area in each frame based on the fire scene during the preprocessing stage, including: FireNet network is used to analyze texture features and color dynamics in images to locate areas of smoke, water vapor and flame interference. The target region is adjusted based on the interference region, and extracted as an image frame by frame; Calculate the proportion of the visible area of ​​the adjusted target region in each frame of the image, and classify the degree of occlusion based on the fire scene according to the proportion of the visible area based on the preset occlusion level classification standard.

6. The firefighter fatigue detection device based on a dynamic region attention mechanism according to claim 4, characterized in that, The adjustment module, in the upsampling layer of the backbone network, adjusts the sampling factor to the same value according to the number of discrete segments, including: Based on the number of discrete segments, the current detection accuracy, and the required detection speed, select a sampling multiple that is the same as the number of discrete segments; By using the sampling factor, the minute visible features in the target area are reconstructed based on the current detection accuracy and detection speed requirements.

Images