Tower crane worker fatigue detection method based on EEG and video signal fusion

Abstract

The invention discloses a tower crane worker fatigue detection method based on EEG and video signal fusion, and relates to the technical field of fatigue detection, and the method comprises the steps: S1, data collection and marking: employing a motion camera to collect face video data of an operator, and employing an EEG sensor to collect EEG signals of the operator; carrying out subjective evaluation on the psychological fatigue of the operator by adopting NASA-TLX scores; s2, data preprocessing: preprocessing the collected face video data and EEG signals respectively, then segmenting the preprocessed video and EEG data by using a sliding window, and ensuring time axis alignment; and S3, multi-modal feature extraction, fusion and fatigue classification. According to the method, automatic extraction and fusion of data features are realized through the neural network, artificial subjective errors are reduced, the efficiency of processing large-scale data is greatly improved, and the complexity of manual feature selection is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue detection, and in particular to a method for detecting fatigue of tower crane workers based on the fusion of EEG and video signals. Background Art

[0002] Common crane worker fatigue detection technologies can be roughly divided into two categories: invasive and non-invasive. Invasive methods usually rely on wearable sensor devices to obtain workers' physiological data, such as fatigue monitoring systems based on EEG signals. Although this method can accurately capture the fatigue state of workers, it usually requires operators to wear special equipment, which increases the complexity and inconvenience of use, especially in actual scenarios such as construction sites.

[0003] To solve this problem, some studies have proposed intelligent seat fatigue detection systems for construction equipment operators, which collect heartbeat and breathing data through optical fiber sensors. However, such systems are often easily interfered by external noise, making it difficult to ensure data stability and detection reliability.

[0004] In contrast, non-invasive methods rely on video signals for monitoring. The most common non-invasive method is a system based on the PERCLOS feature, which assesses fatigue by monitoring the opening and closing of the eyes. However, this method is limited to detecting simple eye movements and cannot identify more complex subtle changes. In addition, changes in lighting conditions can also affect its detection accuracy.

[0005] Therefore, whether it is an invasive method or a non-invasive method, existing fatigue detection methods have problems such as a single data source and susceptibility to environmental influences, making it difficult to provide comprehensive and stable monitoring results. In addition, existing technologies rely on manual feature extraction and data labeling, which is not only prone to subjective errors, but also increases the burden of labeling. At the same time, due to the small size of the data set and the imbalance of the data set caused by individual differences, the existing models are difficult to train effectively when facing more complex problems and cannot accurately predict unknown data. These limitations make the current fatigue detection methods still face many challenges in practical applications. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for detecting fatigue of tower crane workers based on the fusion of EEG and video signals.

[0007] The objective of the present invention is achieved through the following technical solutions: A method for detecting fatigue of tower crane workers based on EEG and video signal fusion, comprising the following steps: S1: Data collection and labeling: A motion camera is used to collect facial video data of the operator, and an EEG sensor is used to collect the operator's EEG signal; the NASA-TLX score is used to conduct a subjective assessment of the operator's mental fatigue; S2: Data preprocessing: The collected facial video data and EEG signals are preprocessed separately, and then the preprocessed video and EEG data are segmented using a sliding window to ensure time axis alignment; the corresponding fatigue state label is assigned to each signal segment according to the NASA-TLX score; S3: Multimodal feature extraction, fusion and fatigue classification, including: S301: inputting the facial video frames after the data preprocessing in step S2 into the 3D convolutional neural network to extract the spatiotemporal features of the video sequence, and then inputting the spatiotemporal feature sequence extracted by the 3D convolutional neural network into the gated recurrent unit network to further capture the long-term variation pattern of fatigue of the tower crane workers; S302: extracting temporal and spatial features from the EEG signal after data preprocessing in step S2 through a multi-scale one-dimensional convolution kernel and a dynamic graph convolution network respectively; S303: fusing the extracted facial long-term change pattern features, EEG time features, and EEG spatial features, and adjusting feature weights using an attention mechanism; S304: Further process through the fully connected layer and output the fatigue state prediction value through the Softmax layer.

[0008] Furthermore, the use of a motion camera to collect facial video data of the operator is specifically that the motion camera is installed on the windshield of the equipment, 0.6 meters away from the operator, with a sampling frequency of 30 frames per second and a resolution of 1440×1440; The use of an EEG sensor to collect the EEG signal of the operator specifically includes using the EEG sensor to collect the EEG signal with a sampling frequency of 256 Hz and supporting 8 independent EEG channels.

[0009] Furthermore, the preprocessing of the collected facial video data and EEG signal specifically includes: Use OpenCV's Haar cascade to detect faces in the collected facial video data. After detecting the faces, use Linknet to accurately segment the face area and resize it to 64×64 pixels. The EEGLAB toolbox was used to preprocess the EEG data, including signal smoothing, signal filtering and data enhancement.

[0010] Furthermore, the 3D convolutional neural network consists of 2 3D convolutional layers, 2 batch normalization layers and 2 average pooling layers, the kernel size of the first 3D convolutional layer is 5×5×5, and the kernel size of the second 3D convolutional layer is 3×3×3.

[0011] Furthermore, the multi-scale one-dimensional convolution kernel is used to process EEG segments in parallel to capture features at different time scales, with larger convolution kernels extracting long-term low-frequency patterns and smaller convolution kernels capturing short-term high-frequency features. Defined as , is the ratio coefficient, is the sampling rate; then, the convolution features are batch normalized and average pooled to generate a temporal feature map.

[0012] Furthermore, the dynamic graph convolutional network is used to extract spatial features of EEG, and the dynamic graph convolutional network includes 3 GCN layers and a convolutional layer with a convolution kernel size of 1×1; the GCN layer is used to extract spatial features from the EEG signal, wherein each EEG electrode is regarded as a node in the graph, and the connection between the electrodes represents the edge of the graph, and the convolutional layer is used to reduce the feature dimension.

[0013] Furthermore, the use of the attention mechanism to adjust the feature weight specifically includes: using a fully connected layer and a Sigmoid function to convert the fused feature into an attention weight ranging from 0 to 1 , and is a trainable parameter, and then the attention weight Multiply by the fused features to enhance the important information in the fused features.

[0014] The beneficial effects of the present invention are: 1) This paper proposes a crane worker fatigue detection method based on EEG and video signal fusion, which realizes automatic extraction and fusion of data features through neural network, which not only reduces human subjective errors, but also greatly improves the efficiency of processing large-scale data and avoids the complexity of manual feature selection.

[0015] 2) The present invention uses the GRU model to capture long-term dependencies in video sequences, which can effectively track the dynamic changes and long-term trends of fatigue and provide support for the study of multi-level fatigue levels.

[0016] 3) This invention uses feature fusion and multimodal data fusion technology to process the interaction between different signals, overcoming the limitations of a single data source. Even in the case of small data sets and unbalanced data, it can achieve better detection results in the face of more complex problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for detecting fatigue of tower crane workers based on EEG and video signal fusion provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1 , the present invention provides a technical solution: A method for detecting fatigue of tower crane workers based on EEG and video signal fusion, comprising the following steps: S1: Data collection and labeling: A motion camera is used to collect facial video data of the operator, and an EEG sensor is used to collect the operator's EEG signal; the NASA-TLX score is used to conduct a subjective assessment of the operator's mental fatigue; In this embodiment, the facial video data of the operator is collected by using a motion camera. Specifically, the motion camera is installed on the windshield of the equipment, 0.6 meters away from the operator, with a sampling frequency of 30 frames per second and a resolution of 1440×1440; The use of an EEG sensor to collect the EEG signal of the operator specifically includes using the EEG sensor to collect the EEG signal with a sampling frequency of 256 Hz and supporting 8 independent EEG channels.

[0020] S2: Data preprocessing: The collected facial video data and EEG signals are preprocessed separately, and then the preprocessed video and EEG data are segmented using a sliding window to ensure time axis alignment; the corresponding fatigue state label is assigned to each signal segment according to the NASA-TLX score; Fatigue state labels can help associate signals in the dataset with specific fatigue levels, thereby better identifying and classifying fatigue.

[0021] In this embodiment, the preprocessing of the collected facial video data and EEG signal specifically includes: The Haar cascade of OpenCV is used to detect faces in the collected facial video data. After the faces are detected, the linknet is used to accurately segment the face area and adjust it to 64×64 pixels. The purpose of readjusting the pixels of the face video frame is to reduce the computational complexity.

[0022] The EEGLAB toolbox was used to preprocess the EEG data, including signal smoothing, signal filtering and data enhancement, to improve data diversity and robustness.

[0023] S3: Multimodal feature extraction, fusion and fatigue classification, including: S301: inputting the facial video frame after the data preprocessing in step S2 into the 3D convolutional neural network to extract the spatiotemporal features of the video sequence, and then inputting the spatiotemporal feature sequence extracted by the 3D convolutional neural network into the gated recurrent unit (GRU) network to further capture the long-term variation pattern of fatigue of the tower crane worker; The 3D convolutional neural network (3D-CNN) consists of 2 3D convolutional layers, 2 batch normalization layers and 2 average pooling layers. The kernel size of the first 3D convolutional layer is 5×5×5, and the kernel size of the second 3D convolutional layer is 3×3×3.

[0024] S302: extracting temporal and spatial features from the EEG signal after data preprocessing in step S2 through a multi-scale one-dimensional convolution kernel and a dynamic graph convolution network respectively; Among them, multi-scale one-dimensional convolution kernel (1D-CNN) is used to process EEG fragments in parallel to capture features of different time scales. Larger convolution kernels extract long-term low-frequency patterns, and smaller convolution kernels capture short-term high-frequency features. The size of the multi-scale one-dimensional convolution kernel is Defined as , is the ratio coefficient, is the sampling rate; then, the convolution features are batch normalized and average pooled to generate a temporal feature map while reducing noise.

[0025] A dynamic graph convolutional network (GCN) is used to extract the spatial features of EEG, and the dynamic graph convolutional network includes 3 GCN layers and a convolutional layer with a convolution kernel size of 1×1; the GCN layer is used to extract spatial features from the EEG signal, where each EEG electrode is regarded as a node in the graph, and the connection between the electrodes represents the edge of the graph. The convolutional layer is used to reduce the feature dimension.

[0026] S303: fusing the extracted facial long-term change pattern features, EEG time features, and EEG spatial features, and adjusting feature weights using an attention mechanism; Furthermore, the use of the attention mechanism to adjust the feature weight specifically includes: using a fully connected layer and a Sigmoid function to convert the fused feature into an attention weight ranging from 0 to 1 , and is a trainable parameter, and then the attention weight Multiply by the fused features to enhance the important information in the fused features.

[0027] S304: Further process through the fully connected layer and output the fatigue state prediction value through the Softmax layer.

[0028] Among them, the Dropout layer is introduced after the fully connected layer and the convolutional layer to reduce the model's dependence on features to prevent overfitting.

[0029] The outputs of the 3D convolution layer, multi-scale one-dimensional convolution kernel, and fully connected layer involved in this embodiment all use the ReLU activation function, which is used to extract nonlinear features, enrich feature information, and then input it into subsequent layers.

[0030] The present invention proposes a method for detecting fatigue of crane workers based on the fusion of EEG and video signals. The automatic extraction and fusion of data features are realized through neural networks, which not only reduces human subjective errors, but also greatly improves the efficiency of processing large-scale data and avoids the complexity of manual feature selection. At the same time, the GRU model is used to capture the long-term dependencies in the video sequence, which can effectively track the dynamic changes and long-term trends of fatigue and provide support for the study of multi-level fatigue levels. In addition, feature fusion and multimodal data fusion technology are used to process the interaction between different signals, overcoming the limitations of a single data source. Even in the case of small data sets and unbalanced data, better detection results can be achieved in the face of more complex problems.

[0031] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A crane worker fatigue detection method based on EEG and video signal fusion, characterized in that: The following steps are involved: S1: Data collection and labeling: A motion camera is used to collect facial video data of the operator, and an EEG sensor is used to collect the operator's EEG signal; the NASA-TLX score is used to conduct a subjective assessment of the operator's mental fatigue; S2: Data preprocessing: Preprocess the collected facial video data and EEG signals respectively, and then use the sliding window to segment the preprocessed video and EEG data to ensure the time axis alignment; Assign a corresponding fatigue status label to each signal segment based on the NASA-TLX score; S3: Multimodal feature extraction, fusion and fatigue classification, including: S301: inputting the facial video frames after the data preprocessing in step S2 into the 3D convolutional neural network to extract the spatiotemporal features of the video sequence, and then inputting the spatiotemporal feature sequence extracted by the 3D convolutional neural network into the gated recurrent unit network to further capture the long-term variation pattern of fatigue of the tower crane workers; S302: extracting temporal and spatial features from the EEG signal after data preprocessing in step S2 through a multi-scale one-dimensional convolution kernel and a dynamic graph convolution network respectively; S303: fusing the extracted facial long-term change pattern features, EEG time features, and EEG spatial features, and adjusting feature weights using an attention mechanism; S304: Further process through the fully connected layer and output the fatigue state prediction value through the Softmax layer.

2. A crane worker fatigue detection method based on EEG and video signal fusion according to claim 1, characterized in that: The method of using a motion camera to collect facial video data of the operator is as follows: the motion camera is installed on the windshield of the equipment, 0.6 meters away from the operator, with a sampling frequency of 30 frames per second and a resolution of 1440×1440; The use of an EEG sensor to collect the EEG signal of the operator specifically includes using the EEG sensor to collect the EEG signal with a sampling frequency of 256 Hz and supporting 8 independent EEG channels.

3. A crane worker fatigue detection method based on EEG and video signal fusion according to claim 1, characterized in that: The preprocessing of the collected facial video data and EEG signal specifically includes: Use OpenCV's Haar cascade to detect faces in the collected facial video data. After detecting the faces, use Linknet to accurately segment the face area and resize it to 64×64 pixels. The EEGLAB toolbox was used to preprocess the EEG data, including signal smoothing, signal filtering and data enhancement.

4. The method for detecting fatigue of crane workers based on EEG and video signal fusion according to claim 1, characterized in that: The 3D convolutional neural network consists of 2 3D convolutional layers, 2 batch normalization layers and 2 average pooling layers. The kernel size of the first 3D convolutional layer is 5×5×5, and the kernel size of the second 3D convolutional layer is 3×3×3.

5. The method for detecting fatigue of crane workers based on EEG and video signal fusion according to claim 1, characterized in that: The multi-scale one-dimensional convolution kernel is used to process EEG segments in parallel and capture features of different time scales. The larger convolution kernel extracts long-term low-frequency patterns, and the smaller convolution kernel captures short-term high-frequency features. The size of the multi-scale one-dimensional convolution kernel is Defined as , is the ratio coefficient, is the sampling rate; then, the convolution features are batch normalized and average pooled to generate a temporal feature map.

6. The method for detecting fatigue of crane workers based on EEG and video signal fusion according to claim 1, characterized in that: The dynamic graph convolutional network is used to extract the spatial features of EEG, and the dynamic graph convolutional network includes 3 GCN layers and a convolutional layer with a convolution kernel size of 1×1; the GCN layer is used to extract spatial features from the EEG signal, wherein each EEG electrode is regarded as a node in the graph, and the connection between the electrodes represents the edge of the graph, and the convolutional layer is used to reduce the feature dimension.

7. The method for detecting fatigue of crane workers based on EEG and video signal fusion according to claim 1, characterized in that: The use of the attention mechanism to adjust the feature weights specifically includes: using a fully connected layer and a Sigmoid function to convert the fused features into an attention weight ranging from 0 to 1 , and is a trainable parameter, and then the attention weight Multiply by the fused features to enhance the important information in the fused features.

Images