Fatigue detection method based on video gait

Abstract

The invention belongs to the cross technical field of computer vision, figure segmentation, fatigue detection and the like, and discloses a video gait-based fatigue detection method, which comprises the following steps of: preprocessing input single fatigue gait and non-fatigue gait videos, extracting a gait contour sequence, passing through a group of continuous CNN (Convolutional Neural Network) network layers, and applying global maximum pooling to obtain a gait contour sequence; and extracting fine-grained spatial features and highlighted spatial features of the contour sequence, fusing the fine-grained spatial features and the highlighted spatial features, and judging whether the human body is in a fatigue state or not. According to the method, the spatial appearance characteristics of each frame are reserved, the gait motion characteristics of pedestrians are fully mined, and the accuracy and robustness of fatigue judgment in a long-distance scene are effectively improved.

Description

Technical Field

[0001] The invention belongs to the cross-technical fields of computer vision, human segmentation, fatigue detection and the like, and specifically relates to a human fatigue detection method based on video gait. Background Art

[0002] Generally, physical fatigue caused by excessive exercise is a common symptom in the field of neuroscience. Human gait is coordinated and regulated by multiple biomechanical factors (such as muscles, joints, and nervous system), which may be affected in a fatigued state, resulting in changes in gait parameters. According to recent research literature, physical fatigue can be assessed by questionnaires or performance of tasks. This study focused on the perception of physical fatigue and defined it by a statistically significant difference from the control group (a group without fatigue).

[0003] In past studies, subjective self-report methods, also known as psychological fatigue assessment, were mainly used, asking participants to assess their physical fatigue based on their own fatigue feelings. Traditional self-report assessments include the use of the Fatigue Severity Scale (FSS), a commonly used tool with good psychometric properties. Another widely used tool is the Rating of Perceived Exertion (RPE), which is used for behavioral and physiological assessment of physical performance and work ability. Although subjective fatigue measurement methods can effectively assess the degree of fatigue in the human body, there are certain challenges in its comprehensive and accurate analysis.

[0004] Another method is objective performance assessment, also known as physiological fatigue assessment, which aims to evaluate the brain function performance of subjects in cognitive tasks. Different tasks can measure different brain performances of participants, and the degree of physical fatigue is expressed by calculating the average reaction time of different cognitive tasks. This method is not affected by subjective factors, so the evaluation results are objective.

[0005] However, both of the above methods require invasive measurements and require subjects to complete questionnaires or cognitive tasks, so it is not convenient to predict physical fatigue in daily life. With the development of smart electronic technology, the application of wearable devices has become more and more common. Some wearable devices can capture multiple physiological indicators at the same time, such as electrocardiogram (ECG), electromyogram (EMG), and galvanic skin response (GSR), which can be used to reliably predict physical fatigue. However, the above gait fatigue assessment methods usually require professional equipment and complex data processing, which limits their popularity and convenience in practical applications.

[0006] With the rapid development of deep learning, it is no longer necessary to wear professional equipment to detect physical fatigue. At present, facial features are widely used to detect human fatigue. These methods model individual behavior by extracting features such as eyes and mouth to monitor their fatigue state.

[0007] The detection methods are relatively mature, but their effects are often poor when the face is occluded. In addition, when the individual is in a long-distance scene, the facial capture device usually cannot clearly capture the facial details, which brings challenges to face-based fatigue detection. Summary of the invention

[0008] Technical problem: In order to solve the above technical problems, the present invention provides a fatigue detection method based on video gait, which can improve the effect of fatigue detection in scenes where the human face is occluded and at a long distance, and has solved the problem of poor fatigue detection effect in scenes where the face is occluded and at a long distance raised in the above background technology.

[0009] Technical solution: In order to achieve the above-mentioned purpose, the present invention is implemented by the following technical solution:

[0010] The present invention is a fatigue detection method based on video gait, comprising the following steps:

[0011] Step 1) Input a video of a single person walking;

[0012] Step 2) The gait video obtained in step 1) is extracted and segmented into a gait profile sequence using the Robust Video Matting algorithm, and the gait profile sequence is processed into a binary gait profile sequence of 64*44 size through preprocessing; the specific steps include:

[0013] Step 21) Using the Robust Video Matting algorithm, the gait video of a single person walking is segmented into binary gait profile sequence graphs, and fatigue / non-fatigue labels are marked;

[0014] Step 22) The binary contour sequence obtained in step 21) is cropped into a size of 64*44 by center alignment;

[0015] Step 23) The gait profile sequence data set and the corresponding image labels obtained in step 22) are divided into a training set and a validation set in a ratio of 8:2.

[0016] Step 3) extracting features of the gait profile sequence obtained in step 2) through a set of continuous CNN networks to extract basic features of the gait profile sequence; the specific steps include:

[0017] Step 31) The 64*44 binary gait profile sequence I∈R extracted in step 2) S×H×W Input to the F1 module, where S represents the number of gait profile frames, H represents the image height, and W represents the image width; the F1 expression is as follows:

[0018] F1=LR(C2(LR(C1)))

[0019] As mentioned above, LR represents the nonlinear activation function Leaky Relu, and C2 and C1 represent convolution;

[0020] Step 32) The F1 module convolves the input binary gait profile sequence and uses the Leaky Relu activation function to obtain the basic gait feature F A ∈R N×S×C×H×W , N is the batch size, C is the number of channels. The input channel is 1, the output channel is 32, the sizes of the two convolution kernels are 5*5 and 3*3, and the step sizes are 2 and 1 respectively.

[0021] Step 4) extracts the maximum frame-level features from the features of the contour sequence obtained in step 3) through the maximum frame-level feature extractor MaxF to extract the salient features in each sample; the specific steps include:

[0022] Step 41) The feature map F extracted in step 3) is A Input into the maximum frame-level feature extractor MaxF to extract the maximum frame-level features; the MaxF module includes MaxF1 and MaxF2 modules, and the MaxF1 expression is as follows:

[0023] Z=G(F A )

[0024] F A = {v i |i=1,2,...,n}

[0025] Z=G{V π(j) |j=1,2,...,n}

[0026] As mentioned above, Z represents the set-level feature, v i represents frame-level features, and π is any permutation and combination;

[0027] Step 42) In a given frame segment, store the maximum value and the maximum value index in each frame segment in a temp list;

[0028] Step 43) Concatenate the maximum value and the maximum value index in the temp list to synthesize the maximum frame-level feature F B ∈R N ×C×H×W ;

[0029] Step 44) The feature F obtained in step 43) B After global maximum pooling, the prominent fatigue gait feature F is obtained C ∈R N×C×H2×W2 , the height and width are both reduced to half of the original;

[0030] Step 45) The MaxF2 module performs two convolutions on the input binary gait profile sequence and uses a nonlinear activation function after the convolution to obtain the feature map F D ∈R N×C×H2×W2 The size of both convolution kernels is 3*3, and the stride is 1. The input channels are 32 and the output channels are 64;

[0031] Step 46) The gait feature graph F extracted in step 45) is D The maximum pooling vector F is obtained by maximum global pooling E ∈R N×C×H4×W4 , the height and width become F D half the height and width;

[0032] Step 47) The feature map F extracted in step 3) is B Input to the MaxF2 module to get the feature map F G ∈R N×C×H2×W2 ;

[0033] Step 48) The feature F extracted in step 47) is G Input to the MaxF1 module to obtain the maximum frame-level feature F H ∈R N ×C×H4×W4 ;

[0034] Step 49) The feature F obtained in step 46) E and the feature F obtained in step 48) H Perform feature fusion to obtain the final output feature F of the MaxF module I ∈R N×C×H4×W4 , the formula is as follows:

[0035] F I =W E ×F E +W H ×F H

[0036] As mentioned above, W E and W H F E and F H The weight of

[0037] Step 5) extracts local spatial features from the features of the contour sequence extracted in step 3) through a fine-grained spatial feature extractor FPFE to obtain fine-grained spatial features in each sample; the specific steps include:

[0038] Step 51) The gait feature F obtained in step 3) is BInput into the fine-grained spatial feature extractor FPFE to extract the fine-grained spatial features F of each frame image O . F O The formula is as follows:

[0039] F O =FPFE(f k )

[0040] S F ={F O |k=1,2,...,n}

[0041] As described above, k=1,2,...,n represents the index of the frame in the gait sequence, S F is the feature graph sequence, F O is a four-dimensional tensor, i.e., batch size, number of channels, height, and width. FPFE consists of three modules, each of which includes two convolutional layers;

[0042] Step 52) Input the input feature map into the first module Block1 for conventional convolution operation to obtain the feature map F J ∈R N×C×H×W Block1 includes two convolution kernels of size 5*5 and 3*3, with 1 input channel and 32 output channels;

[0043] Step 53) The feature F extracted in step 52) is J The maximum pooling vector F is obtained by maximum global pooling K ∈R N ×C×H2×W2 ;

[0044] Step 54) The feature vector F extracted in step 53) is K Input to the second module Block2, Block2 contains two convolution kernels of size 3*3 and a LeakyReLU activation function. The LeakyRelu formula is as follows:

[0045]

[0046] Step 55) The feature F extracted in step 53) is K The image is divided into four parts horizontally, and two convolution operations are performed on each of the four parts, with 32 input channels and 64 output channels, and the LeakyReLU activation function is applied to obtain the feature map F. L ∈R N ×C×H2×W2 ;

[0047] Step 56) The feature map F extracted in step 55) is L The maximum pooling vector F is obtained by global maximum pooling M ∈RN ×C×H4×W4 ;

[0048] Step 57) The feature vector F extracted in step 56) is M Input to the third module Block3, which contains two convolution kernels of size 3*3 and a LeakyReLU activation function;

[0049] Step 58) The feature F extracted in step 56) is M The horizontal segmentation is divided into 8 parts, and two convolution operations are performed on each of the 8 parts. The input channel and output channel are both 64, and the LeakyReLU activation function is applied to obtain the feature map F N ∈R N ×C×H4×W4 ;

[0050] Step 59) The feature F extracted in step 58) is N The 8 parts of are horizontally cascaded to obtain feature F O , as the final output feature F of FPFE O ;

[0051] Step 6) The prominent fatigue gait features obtained in step 4) and the fine-grained fatigue gait spatial features obtained in step 5) are fused based on the SE attention mechanism to obtain the final fatigue gait features; the specific steps include:

[0052] Step 61) The prominent feature F obtained in step 4) is I And the fine-grained spatial feature F obtained in step 5) O , through linear change and a Sigmoid function, calculate F I and F O The attention weight is denoted as W I and W O , the formula is as follows:

[0053]

[0054] As described above, W is the weight matrix automatically learned by the model, and b is the bias, which is used to adjust the original score of the linear transformation.

[0055] Step 62) The attention weight W obtained in step 61) is I and the prominent feature F obtained in step 4) I Perform a multiplication operation. The attention weight W obtained in step 61) O And the fine-grained spatial feature F obtained in step 5) O Perform a multiplication operation and perform feature fusion based on the SE attention mechanism to obtain the final fatigue gait feature F P ∈RN×C×H4×W4 :

[0056] F P =W I ×F I +W O ×F O ;

[0057] Step 7) The fatigue gait features obtained in step 6) are subjected to fatigue classification through linear transformation, and the final result is output; the specific steps include:

[0058] Step 71) According to the feature F obtained in step 62) P , apply Linear1 and Linear2 two linear transformations, and apply ReLU function after Linear1 and Linear2 to get the output feature vector F X1 and F X2 , the formula is as follows:

[0059] F X1 =max(0,(W*F P +b))

[0060] F X2 =max(0,(W*F X1 + b));

[0061] Step 72) According to the F obtained in step 71) X2 , apply Linear3 linear transformation to obtain the final output feature vector F OUT , and for F OUT The Sigmoid function is used to map the output of the model to the range of [0,1] to determine whether the human body is in a state of fatigue. The formula is as follows:

[0062]

[0063] Beneficial effects: The present invention first extracts a gait contour sequence from a video of a single person walking, and extracts the basic features of the contour sequence from it. Secondly, the salient spatial features and fine-grained spatial features are obtained through the maximum frame-level feature extractor and the fine-grained spatial feature extractor respectively, and feature fusion is performed based on the SE attention mechanism. Finally, the fatigue detection result is output through linear transformation and activation function. This method can effectively improve the accuracy of fatigue detection in scenes with facial occlusion and long distance, and cope with the challenges brought by practical applications.

[0064] Specifically:

[0065] (1) The present invention uses the Robust Video Matting algorithm to extract the gait profile sequence of a single person video as the input representation of the pedestrian fatigue gait.

[0066] (2) The present invention extracts basic features through a set of continuous CNN networks, and extracts salient spatial features through a maximum frame-level feature extractor. Fine-grained spatial features are extracted through a fine-grained spatial feature extractor, and the fine-grained spatial feature extractor includes three modules, each of which includes two convolutional layers.

[0067] (3) The present invention uses the SE attention mechanism to fuse the salient spatial features with the fine-grained spatial features to obtain the final fatigue gait features.

[0068] (4) The present invention performs a linear transformation on the final fatigue gait feature and uses an activation function to map the output of the model to the range of [0, 1] to determine whether the human body is in a fatigue state. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of the fatigue detection method based on video gait of the present invention.

[0070] Figure 2 It is an architecture diagram of the fatigue detection network based on video gait of the present invention. DETAILED DESCRIPTION

[0071] The following will describe the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary.

[0072] As a biological characteristic of human movement, gait contains rich information and can reflect the individual's physical condition and fatigue level. By analyzing and mining gait data, valuable fatigue indicators can be obtained to provide a scientific basis for fatigue monitoring and prevention. For example, segmentation algorithms can be used to process and analyze the collected video gait, and by learning the prominent spatial features and fine-grained spatial features of gait and performing feature fusion, it can be used to determine whether the human body is in a state of fatigue.

[0073] like Figure 1 , Figure 2 As shown, the present invention is a gait recognition method based on adaptive fusion to generate multi-scale time features, comprising the following steps:

[0074] Step 1) Input a video of a single person walking.

[0075] Step 2) Use the Robust Video Matting algorithm to extract and segment the single-person walking image in the gait video obtained in step 1) into a gait contour sequence, and through preprocessing, process the gait contour sequence into a binary gait contour sequence of 64*44 size.

[0076] The gait profile sequence extraction comprises the following steps:

[0077] Step 21) Using the Robust Video Matting algorithm, the gait video of a single person walking is segmented into binary gait profile sequence graphs, and fatigue / non-fatigue labels are marked;

[0078] Step 22) The binary contour sequence obtained in step 21) is cropped into a size of 64*44 by center alignment;

[0079] Step 23) The gait profile sequence data set and the corresponding image labels obtained in step 22) are divided into a training set and a validation set in a ratio of 8:2.

[0080] Step 3) extracts features of the gait profile sequence obtained in step 2) through a set of continuous CNN networks to extract basic features of the gait profile sequence.

[0081] The basic feature extraction of gait profile sequence comprises the following steps:

[0082] Step 31) The 64*44 binary gait profile sequence I∈R extracted in step 2) S×H×W Input to the F1 module, where S represents the number of gait profile frames, H represents the image height, and W represents the image width; the F1 expression is as follows:

[0083] F1=LR(C2(LR(C1)))

[0084] As mentioned above, LR represents the nonlinear activation function Leaky Relu, and C2 and C1 represent convolution;

[0085] Step 32) The F1 module convolves the input binary gait profile sequence and uses the Leaky Relu activation function to obtain the basic gait feature F A ∈R N×S×C×H×W , N is the batch size, C is the number of channels. The input channel is 1, the output channel is 32, the sizes of the two convolution kernels are 5*5 and 3*3, and the step sizes are 2 and 1 respectively.

[0086] Step 4) The features of the contour sequence obtained in step 3) are extracted by a maximum frame-level feature extractor MaxF to extract the maximum frame-level features, so as to extract the salient spatial features in each sample.

[0087] The extraction of the prominent spatial features of fatigue gait includes the following steps:

[0088] Step 41) The feature map F extracted in step 3) is A Input into the maximum frame-level feature extractor MaxF to extract the maximum frame-level features; the MaxF module includes MaxF1 and MaxF2 modules, and the MaxF1 expression is as follows:

[0089] Z=G(F A )

[0090] F A = {v i |i=1,2,...,n}

[0091] Z=G{V π(j) |j=1,2,...,n}

[0092] As mentioned above, Z represents the set-level feature, v i represents frame-level features, and π is any permutation and combination;

[0093] Step 42) In a given frame segment, store the maximum value and the maximum value index in each frame segment in a temp list;

[0094] Step 43) Concatenate the maximum value and the maximum value index in the temp list to synthesize the maximum frame-level feature F B ∈R N ×C×H×W ;

[0095] Step 44) The feature F obtained in step 43) B After global maximum pooling, the prominent fatigue gait feature F is obtained C ∈R N×C×H2×W2 , the height and width are both reduced to half of the original;

[0096] Step 45) The MaxF2 module performs two convolutions on the input binary gait profile sequence and uses a nonlinear activation function after the convolution to obtain the feature map F D ∈R N×C×H2×W2 The size of both convolution kernels is 3*3, and the stride is 1. The input channels are 32 and the output channels are 64;

[0097] Step 46) The gait feature graph F extracted in step 45) is D The maximum pooling vector F is obtained by maximum global pooling E ∈R N×C×H4×W4 , the height and width become F D half the height and width;

[0098] Step 47) The feature map F extracted in step 3) is B Input to the MaxF2 module to get the feature map F G ∈R N×C×H2×W2 ;

[0099] Step 48) The feature F extracted in step 47) is G Input to the MaxF1 module to obtain the maximum frame-level feature F H ∈R N ×C×H4×W4 ;

[0100] Step 49) The feature F obtained in step 46) E and the feature F obtained in step 48) H Perform feature fusion to obtain the final output feature F of the MaxF module I ∈R N×C×H4×W4 , the formula is as follows:

[0101] F I =W E ×F E +W H ×F H

[0102] As mentioned above, W E and W H F E and F H The weight of

[0103] Step 5) The features of the contour sequence extracted in step 3) are extracted using a fine-grained spatial feature extractor FPFE to extract local spatial features to obtain fine-grained spatial features in each sample.

[0104] The extraction of fine-grained spatial features includes the following steps:

[0105] Step 51) The gait feature F obtained in step 3) is B Input into the fine-grained spatial feature extractor FPFE to extract the fine-grained spatial features F of each frame image O . F O The formula is as follows:

[0106] F O =FPFE(f k )

[0107] S F ={F O |k=1,2,...,n}

[0108] As described above, k=1,2,...,n represents the index of the frame in the gait sequence, S F is the feature graph sequence, F Ois a four-dimensional tensor, i.e., batch size, number of channels, height, and width. FPFE consists of three modules, each of which includes two convolutional layers;

[0109] Step 52) Input the input feature map into the first module Block1 for conventional convolution operation to obtain the feature map F J ∈R N×C×H×W Block1 includes two convolution kernels of size 5*5 and 3*3, with 1 input channel and 32 output channels;

[0110] Step 53) The feature F extracted in step 52) is J The maximum pooling vector F is obtained by maximum global pooling K ∈R N ×C×H2×W2 ;

[0111] Step 54) The feature vector F extracted in step 53) is K Input to the second module Block2, Block2 contains two convolution kernels of size 3*3 and a LeakyReLU activation function. The LeakyRelu formula is as follows:

[0112]

[0113] Step 55) The feature F extracted in step 53) is K The image is divided into four parts horizontally, and two convolution operations are performed on each of the four parts, with 32 input channels and 64 output channels, and the LeakyReLU activation function is applied to obtain the feature map F. L ∈R N ×C×H2×W2 ;

[0114] Step 56) The feature map F extracted in step 55) is L The maximum pooling vector F is obtained by global maximum pooling M ∈R N ×C×H4×W4 ;

[0115] Step 57) The feature vector F extracted in step 56) is M Input to the third module Block3, which contains two convolution kernels of size 3*3 and a LeakyReLU activation function;

[0116] Step 58) The feature F extracted in step 56) is M The horizontal segmentation is divided into 8 parts, and two convolution operations are performed on each of the 8 parts. The input channel and output channel are both 64, and the LeakyReLU activation function is applied to obtain the feature map F N ∈R N×C×H4×W4 ;

[0117] Step 59) The feature F extracted in step 58) is N The 8 parts of are horizontally cascaded to obtain feature F O , as the final output feature F of FPFE O ;

[0118] Step 6) The prominent fatigue gait features obtained in step 4) and the fine-grained fatigue gait spatial features obtained in step 5) are fused based on the SE attention mechanism to obtain the final fatigue gait features.

[0119] As described above, the fusion of salient features and fine-grained spatial features includes the following steps:

[0120] Step 61) The prominent feature F obtained in step 4) is I And the fine-grained spatial feature F obtained in step 5) O , through linear change and a Sigmoid function, calculate F I and F O The attention weight is denoted as W I and W O , the formula is as follows:

[0121]

[0122] As described above, W is the weight matrix automatically learned by the model, and b is the bias, which is used to adjust the original score of the linear transformation.

[0123] Step 62) The attention weight W obtained in step 61) is I and the prominent feature F obtained in step 4) I Perform a multiplication operation. The attention weight W obtained in step 61 is O And the fine-grained spatial feature F obtained in step 5) O Perform a multiplication operation and perform feature fusion based on the SE attention mechanism to obtain the final fatigue gait feature F P ∈R N×C×H4×W4 :

[0124] F P =W I ×F I +W O ×F O ;

[0125] Step 7) performing fatigue classification on the fatigue gait features obtained in step 6) through linear transformation, and outputting the final result;

[0126] The determination of fatigue state includes the following steps:

[0127] Step 71) According to the feature F obtained in step 6) P , apply Linear1 and Linear2 two linear transformations, and apply ReLU function after Linear1 and Linear2 to get the output feature vector F X1 and F X2 , the formula is as follows:

[0128] F X1 =max(0,(W*F P +b))

[0129] F X2 =max(0,(W*F X1 + b));

[0130] Step 72) According to the F obtained in step 71) X2 , apply Linear3 linear transformation to obtain the final output feature vector F OUT , and for F OUT The sigmoid function is used to map the output of the model to the range of [0,1] to determine whether the human body is in a state of fatigue. The formula is as follows:

[0131]

[0132] The present invention can well preserve the gait contour features of each frame, effectively learn the salient spatial features and fine-grained spatial features of the gait, fully explore the fatigue gait motion characteristics of pedestrians, and effectively improve the accuracy and robustness of fatigue detection in long-distance scenarios.

Claims

1. A fatigue detection method based on video gait, characterized in that: The fatigue detection method comprises the following steps: Step 1) Input a video of a single person walking; Step 2) using a Robust Video Matting algorithm to extract and segment the single person walking image in the gait video into a gait profile sequence, and through preprocessing, processing the gait profile sequence into a binary gait profile sequence; Step 3) extracting features of the binary gait profile sequence obtained in step 2) through a set of continuous CNN networks to extract basic features of the binary gait profile sequence; Step 4) extracting the maximum frame-level features from the features of the contour sequence obtained in step 3) through a maximum frame-level feature extractor MaxF to extract the salient features in each sample; Step 5) extracting local spatial features from the features of the contour sequence extracted in step 3) through a fine-grained spatial feature extractor FPFE to obtain fine-grained spatial features in each sample; Step 6) performing feature fusion based on the SE attention mechanism on the prominent fatigue gait features obtained in step 4) and the fine-grained fatigue gait spatial features obtained in step 5) to obtain the final fatigue gait features; Step 7) The fatigue gait features obtained in step 6) are subjected to fatigue classification through linear transformation, and the final result is output.

2. The method for fatigue detection based on video gait according to claim 1, characterized in that: The specific steps of step 2) include: Step 21) Using the Robust Video Matting algorithm, the gait video of a single person walking is segmented into binary gait profile sequence graphs, and fatigue / non-fatigue labels are marked; Step 22) The binary gait profile sequence obtained in step 21) is cropped by center alignment; Step 23) divides the binary gait profile sequence data set and the corresponding image labels obtained in step 22) into a training set and a validation set.

3. The method for fatigue detection based on video gait according to claim 1, characterized in that: The specific steps of step 3) include: Step 31) The binary gait profile sequence I∈R extracted in step 2) is S×H×W Input to the F1 module, S represents the number of gait profile frames, H represents the image height, and W represents the image width; the F1 expression is as follows: F1=LR(C2(LR(C1))) LR represents the nonlinear activation function LeakyRelu, C2 and C1 represent convolution; Step 32) The F1 module convolves the input binary gait profile sequence and uses the LeakyRelu activation function to obtain the basic gait feature F A ∈R N×S×C×H×W , N is the batch size, and C is the number of channels.

4. The method for fatigue detection based on video gait according to claim 1, characterized in that: The specific steps of step 4) include: Step 41) The feature map F extracted in step 3) is A Input into the maximum frame-level feature extractor MaxF to extract the maximum frame-level features; the MaxF module includes MaxF1 and MaxF2 modules, and the MaxF1 expression is as follows: Z=G(F A ) F A ={v i |i=1,2,...,n} Z=G{V π(j) |j=1,2,...,n} As mentioned above, Z represents the set-level feature, v i represents the frame-level feature, π is any permutation and combination, and n is the index of the frame in the gait sequence; Step 42) In a given frame segment, store the maximum value and the maximum value index in each frame segment in a temp list; Step 43) Concatenate the maximum value and the maximum value index in the temp list to synthesize the maximum frame-level feature F B ∈R N ×C×H×W ; Step 44) The feature F obtained in step 43) B After global maximum pooling, the prominent fatigue gait feature F is obtained C ∈R N ×C×H / 2×W / 2 , the height and width are both reduced to half of the original; Step 45) The MaxF2 module performs two convolutions on the input binary gait profile sequence and uses a nonlinear activation function after the convolution to obtain the feature map F D ∈R N×C×H / 2×W / 2 ; Step 46) The gait feature graph F extracted in step 45) is D The maximum pooling vector F is obtained by maximum global pooling E ∈R N ×C×H / 4×W / 4 , the height and width become F D half the height and width; Step 47) The feature map F extracted in step 3) is B Input to the MaxF2 module to get the feature map F G ∈R N×C×H / 2×W / 2 ; Step 48) The feature F extracted in step 47) is G Input to the MaxF1 module to obtain the maximum frame-level feature F H ∈R N×C×H / 4×W / 4 ; Step 49) The feature F obtained in step 46) E and the feature F obtained in step 48) H Perform feature fusion to obtain the final output feature F of the MaxF module I ∈R N×C×H / 4×W / 4 , the formula is as follows: F I =W E ×F E +W H ×F H As mentioned above, W E and W H F E and F H The weight of .

5. The method for fatigue detection based on video gait according to claim 1, characterized in that: The specific steps of step 5) include: Step 51) The gait feature F obtained in step 3) is B Input into the fine-grained spatial feature extractor FPFE to extract the fine-grained spatial features F of each frame image O , F O The formula is as follows: F O =FPFE(f k ) S F ={F O |k=1,2,...,n} k=1,2,...,n represents the index of the frame in the gait sequence, f x is the gait feature graph sequence, F O is a four-dimensional tensor, i.e., batch size, number of channels, height, and width; FPFE consists of three modules, each of which includes two convolutional layers; Step 52) Input the input feature map into the first module Block1 for conventional convolution operation to obtain the feature map F J ∈R N×C×H×W ; Step 53) The feature F extracted in step 52) is J The maximum pooling vector F is obtained by maximum global pooling K ∈R N×C×H / 2×W / 2 ; Step 54) The feature vector F extracted in step 53) is K Input to the second module Block2, Block2 contains two convolution kernels and a LeakyReLU activation function. The LeakyRelu formula is as follows: Step 55) The feature F extracted in step 53) is K The image is divided into four parts horizontally, and two convolution operations are performed on each of the four parts. The LeakyReLU activation function is applied to obtain the feature map F. L ∈R N×C×H / 2×W / 2 ; Step 56) The feature map F extracted in step 55) is L The maximum pooling vector F is obtained by global maximum pooling M ∈R N ×C×H / 4×W / 4 ; Step 57) The feature vector F extracted in step 56) is M Input to the third module Block3, which contains two convolution kernels and a LeakyReLU activation function; Step 58) The feature F extracted in step 56) is M The horizontal segmentation is divided into 8 parts, and the 8 parts are convolved twice, and the LeakyReLU activation function is applied to obtain the feature map F N ∈R N×C×H / 4×W / 4 ; Step 59) The feature F extracted in step 58) is N The 8 parts of are horizontally cascaded to obtain feature F O , as the final output feature F of FPFE O .

6. The method for fatigue detection based on video gait according to claim 1, characterized in that: The specific steps of step 6) include: Step 61) The prominent feature F obtained in step 4) is I And the fine-grained spatial feature F obtained in step 5) O , through linear change and a Sigmoid function, calculate F I and F O The attention weight is denoted as W I and W O , the formula is as follows: As described above, W is the weight matrix automatically learned by the model, and b is the bias, which is used to adjust the original score of the linear transformation; Step 62) The attention weight W obtained in step 61) is I and the prominent feature F obtained in step 4) I Perform multiplication operation; the attention weight W obtained in step 61) O And the fine-grained spatial feature F obtained in step 5) O Perform a multiplication operation and perform feature fusion based on the SE attention mechanism to obtain the final fatigue gait feature F P ∈R N×C×H / 4×W / 4 : F P =W I ×F I +W O ×F O 。 7. The method for fatigue detection based on video gait according to claim 6, characterized in that: The specific steps of step 7) include: Step 71) According to the feature F obtained in step 62) P , apply Linear1 and Linear2 two linear transformations, and apply ReLU function after Linear1 and Linear2 to get the output feature vector F X1 and F X2 , the formula is as follows: F X1 =max(0,(W*F P +b)) F X2 =max(0,(W*F X1 +b)); Step 72) According to the F obtained in step 71) X2 , apply Linear3 linear transformation to obtain the final output feature vector F OUT , and for F OUT The Sigmoid function is used to map the output of the model to the range of [0,1] to determine whether the human body is in a state of fatigue. The formula is as follows:

Images