Seaman workload assessment method and system based on deep learning

Abstract

The invention provides a crew workload evaluation method and system based on deep learning, and relates to the technical field of navigation safety monitoring, and the method comprises the steps: collecting an electroencephalogram signal and operation behavior data of a crew in a target navigation scene, and carrying out the preprocessing of the electroencephalogram signal; extracting power features of a plurality of frequency bands in the preprocessed electroencephalogram signals, calculating combined indexes related to workloads, and constructing a multi-dimensional feature set in combination with a target navigation scene and corresponding operation behavior data; constructing a deep learning model containing a convolutional neural network and a bidirectional long-short-term memory network, and training, optimizing and evaluating the deep learning model by taking the multi-dimensional feature set as input and the workload level as output; and inputting a sailor electroencephalogram signal which is acquired and preprocessed in real time into the trained deep learning model, outputting a current workload level, and triggering graded early warning according to the current workload level and an operation behavior corresponding to the current navigation scene.

Description

Technical Field

[0001] This invention relates to the field of maritime safety monitoring technology, and in particular to a method and system for assessing crew workload based on deep learning. Background Technology

[0002] In high-risk maritime operations, human factors are one of the main causes of accidents, and the workload of crew members directly affects their cognitive abilities and operational performance. Both excessively high and low workloads reduce operational efficiency and increase the risk of operational errors; therefore, scientific assessment of workload is of great importance.

[0003] Existing workload assessment methods are mainly divided into three categories: subjective assessment, behavioral assessment, and physiological assessment. Subjective assessment relies on workers' self-reports, such as various scale ratings, which are subject to lag and are easily influenced by subjective biases. Behavioral assessment indirectly infers workload status by analyzing operational performance data, but it is difficult to distinguish the influence of workload on other factors such as skill level. Physiological assessment is based on the analysis of human physiological signals, and has the advantages of strong objectivity and good real-time performance, making it a current research hotspot. Electroencephalogram (EEG) signals, as physiological signals that directly reflect the brain's cognitive activities, contain rich information on neural activity, and fluctuations in different frequency bands are closely related to cognitive processes such as attention, memory, and decision-making.

[0004] However, existing assessment methods based on EEG signals lack feature systems and assessment models specific to maritime scenarios. Traditional machine learning models struggle to effectively capture the complex spatiotemporal dynamics and long-term temporal dependencies in EEG signals, resulting in limited assessment accuracy and robustness. It is difficult to accurately correlate and fuse fluctuations in neurophysiological indicators with specific ship handling events and changes in environmental parameters, leading to poor accuracy, real-time performance, and scenario adaptability. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based method and system for assessing crew workload, in order to solve the problems of poor accuracy, real-time performance, and scene adaptability of existing assessment methods based on electroencephalogram (EEG) signals mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a deep learning-based method for assessing crew workload, comprising the following steps: collecting electroencephalogram (EEG) signals and operational behavior data of crew members in a target maritime scenario, and preprocessing the EEG signals; extracting power features of multiple frequency bands from the preprocessed EEG signals, and calculating combined indicators related to workload, and constructing a multi-dimensional feature set by combining the target maritime scenario and corresponding operational behavior data; constructing a deep learning model including a convolutional neural network and a bidirectional long short-term memory network, using the multi-dimensional feature set as input and the workload level as output, and training, optimizing, and evaluating the deep learning model; inputting the real-time collected and preprocessed crew EEG signals into the trained deep learning model, outputting the current workload level, and triggering graded early warnings based on the current workload level and the operational behavior corresponding to the current maritime scenario.

[0007] Optionally, the target navigation scenario is constructed using a ship simulator, specifically including at least one of the following: open water cruising, narrow channel encounters, low visibility fog navigation avoidance, emergency response to personnel falling into the water, and emergency handling of steering gear malfunctions; the operational behavior data includes whether to adjust the rudder angle, accelerate or decelerate the speed, and / or whether to take collision avoidance actions; the multi-dimensional feature set includes EEG signals, the target navigation scenario, operational behavior data, and combined indicators related to workload.

[0008] Optionally, when constructing a multi-dimensional feature set, a sliding window method is used to process the power features and the combined indicators to generate a time-series feature matrix. The width of the sliding window is 2 to 10 seconds, and the step size is 0.5 to 2 seconds. The row dimension of the time-series feature matrix is ​​the time point, and the column dimension is the feature parameter.

[0009] Optionally, the preprocessing step of the EEG signal specifically includes: bandpass filtering the EEG signal using a finite impulse response filter or an infinite impulse response filter, with a filtering frequency band of 0.1Hz-100Hz, to retain the main cognitive-related frequency bands in the EEG signal while removing DC drift and high-frequency noise; identifying and processing motion artifacts and / or physiological artifacts in the EEG signal using independent principal component analysis combined with manual inspection, and repairing the identified artifact segments using cubic spline interpolation or neighboring data replacement; aligning the EEG signal, operational behavior data, and target nautical scene using scene timestamps to form a time-series dataset, and standardizing the time-series dataset to eliminate signal amplitude deviations caused by individual differences.

[0010] Optionally, the step of extracting power features of multiple frequency bands in the preprocessed EEG signal and calculating the combined index related to workload specifically includes: performing spectral analysis on the EEG signal using Fourier transform or wavelet transform, decomposing the EEG signal into five frequency bands: delta, theta, alpha, beta, and gamma, and calculating the power spectral density or average power of each frequency band; and calculating the power ratio or weighted combination value of different frequency bands based on the power spectral density or average power of each frequency band.

[0011] Optionally, the formulas for calculating the power ratios of different frequency bands are: Ratio1 = PSD_θ / PSD_α; where: PSD_θ is the power spectral density of the theta band, and PSD_α is the power spectral density of the alpha band; Ratio2 = PSD_β / (PSD_α + PSD_θ); where: PSD_β is the power spectral density of the beta band, PSD_α is the power spectral density of the alpha band, and PSD_θ is the power spectral density of the theta band; Ratio3 = PSD_γ / (PSD_θ + PSD_α); where: PSD_γ is the power spectral density of the gamma band, PSD_θ is the power spectral density of the theta band, and PSD_α is the power spectral density of the alpha band; Ratio4 = PSD_β / PSD_θ; where: PSD_β is the power spectral density of the beta band, and PSD_θ is the power spectral density of the theta band; the formula for calculating the weighted combination value of the frequency band power is: In the formula: These are the weighting coefficients for each frequency band. Let be the power spectral density of each frequency band, where the weight of each frequency band is determined by principal component analysis.

[0012] Optionally, the step of constructing a deep learning model comprising a convolutional neural network and a bidirectional long short-term memory network specifically includes: the convolutional neural network extracts local spectral features through a one-dimensional convolutional kernel and reduces the dimensionality through a pooling layer; the bidirectional long short-term memory network captures the temporal dependencies of feature sequences through memory unit state updates; and the model output is a workload level indicating low or high workload.

[0013] Optionally, the steps of training, optimizing, and evaluating the deep learning model specifically include: training the deep learning model by optimizing parameters through minimizing the loss function; optimizing the deep learning model using an early stopping strategy, stopping training when the validation set loss does not decrease for 5 consecutive epochs; and evaluating the deep learning model using accuracy, area under the ROC curve, precision, recall metrics, combined with cross-validation, case validation, and axiomatic validation.

[0014] Optionally, the triggering conditions for the graded warning include: if the high load probability output by the deep learning model lasts for 10 seconds and the Ratio1 ratio increases by 30%~50% from the baseline or the Ratio2 ratio increases for 10 seconds, a first-level warning is triggered; if the high load probability output by the deep learning model lasts for 30 seconds and the Ratio3 ratio increases by ≥30% from the baseline or the Ratio4 ratio increases for ≥30%, and the collision avoidance reaction time is at the scene average level, a second-level warning is triggered.

[0015] On the other hand, the present invention also provides a deep learning-based crew workload assessment system, comprising: a data acquisition module for acquiring crew members' EEG signals and operational behavior data in a target maritime scenario, and preprocessing the EEG signals; a feature set construction module for extracting power features of multiple frequency bands from the preprocessed EEG signals, calculating combined indicators related to workload, and constructing a multi-dimensional feature set by combining the target maritime scenario and corresponding operational behavior data; a model construction module for constructing a deep learning model including a convolutional neural network and a bidirectional long short-term memory network, using the multi-dimensional feature set as input and the workload level as output, and training, optimizing, and evaluating the deep learning model; and an early warning module for inputting the real-time acquired and preprocessed crew members' EEG signals into the trained deep learning model, outputting the current workload level, and triggering graded early warnings based on the current workload level and the operational behavior corresponding to the current maritime scenario.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This application constructs a comprehensive data acquisition framework covering internal factors, external factors, and behavioral performance by synchronously collecting EEG signals and operational behavior data of crew members in a target maritime scenario within a ship simulator. By extracting frequency band power features related to cognitive activities from the EEG signals and further calculating combined indicators, this data is ultimately integrated with behavioral and scenario information. This transforms the raw, low-level signal data into high-level, multi-dimensional fusion features with clear cognitive meaning, revealing more profound neurophysiological mechanisms of workload and its correlation with external tasks and behaviors. Furthermore, by constructing and training a deep learning model integrating convolutional neural networks and bidirectional long short-term memory networks, leveraging the former's powerful local feature extraction capabilities and the latter's ability to model temporal dynamics, an intelligent assessment model is formed that can adaptively learn the characteristics of maritime scenarios and accurately capture the temporal variation patterns of workload. This model instantly outputs workload levels and activates corresponding early warning mechanisms based on specific scenario risks, significantly improving the accuracy, real-time performance, and scenario adaptability of crew workload assessment. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method steps of the present invention.

[0018] Figure 2 This is a flowchart of the method of the present invention.

[0019] Figure 3 This is a schematic diagram of the system structure of the present invention.

[0020] In the diagram: 10 - data acquisition module, 20 - feature set construction module, 30 - model construction module, 40 - early warning module. Detailed Implementation

[0021] The present invention will now be clearly and completely described in conjunction with the accompanying drawings of the embodiments thereof. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0024] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0025] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] Please refer to Figure 1 and Figure 2 This invention discloses a deep learning-based method for assessing crew workload, comprising the following steps: collecting electroencephalogram (EEG) signals and operational behavior data of crew members in a target maritime scenario, and preprocessing the EEG signals; extracting power features of multiple frequency bands from the preprocessed EEG signals, calculating combined indicators related to workload, and constructing a multi-dimensional feature set by combining the target maritime scenario and corresponding operational behavior data; constructing a deep learning model including a convolutional neural network and a bidirectional long short-term memory network, using the multi-dimensional feature set as input and the workload level as output, and training, optimizing, and evaluating the deep learning model; inputting the real-time collected and preprocessed crew EEG signals into the trained deep learning model, outputting the current workload level, and triggering graded early warnings based on the current workload level and the operational behavior corresponding to the current maritime scenario.

[0028] Specifically, in the data collection phase, the first step was the recruitment and screening of participants. The recruitment criteria were 11 male maritime workers aged 25-45 years old, holding a Class A captain's certificate; those with a history of epilepsy, mental illness, recent use of psychotropic medications, or a history of head injury were excluded. This experiment has been approved by the ethics committee, and all participants signed informed consent forms.

[0029] A single-channel wireless EEG acquisition device was used, with a sampling rate set to 512Hz. The electrodes were placed in the prefrontal cortex region, and the reference electrode was placed in the earlobe.

[0030] Ship simulator: Equipped with a 180° visual system, realistic ship control devices, and an environmental sound simulation system.

[0031] Behavior recording devices: High-definition cameras record operational actions, and the simulator's built-in data recording module records environmental parameters.

[0032] This application constructs a comprehensive data acquisition framework covering internal factors, external factors, and behavioral performance by synchronously collecting EEG signals and operational behavior data of crew members in a target maritime scenario within a ship simulator. By extracting frequency band power features related to cognitive activities from the EEG signals and further calculating combined indicators, this data is ultimately integrated with behavioral and scenario information. This transforms the raw, low-level signal data into high-level, multi-dimensional fusion features with clear cognitive meaning, revealing more profound neurophysiological mechanisms of workload and its correlation with external tasks and behaviors. Furthermore, by constructing and training a deep learning model integrating convolutional neural networks and bidirectional long short-term memory networks, leveraging the former's powerful local feature extraction capabilities and the latter's ability to model temporal dynamics, an intelligent assessment model is formed that can adaptively learn the characteristics of maritime scenarios and accurately capture the temporal variation patterns of workload. This model instantly outputs workload levels and activates corresponding early warning mechanisms based on specific scenario risks, significantly improving the accuracy, real-time performance, and scenario adaptability of crew workload assessment.

[0033] In some embodiments, the target navigation scenario is constructed using a ship simulator, specifically including at least one of open water cruising, narrow channel encounters, low visibility fog navigation avoidance, emergency response to personnel falling into the water, and emergency handling of steering gear failure; the operational behavior data includes whether to adjust the rudder angle, accelerate or decelerate the speed, and / or whether to take collision avoidance actions; the multi-dimensional feature set includes EEG signals, target navigation scenario, operational behavior data, and combined indicators related to workload.

[0034] Specifically, in the preparation phase: subjects wear EEG devices, undergo adaptive training, and become familiar with simulator operation.

[0035] Baseline acquisition: Subjects sat still with their eyes closed for 5 seconds to collect baseline EEG signals.

[0036] Task execution: Participants completed a 30-minute simulated flight mission, including the following scenarios: Scenario 1: Normal navigation in open water.

[0037] Scenario 2: Navigating a narrow waterway + encountering other ships.

[0038] Scenario 3: Low visibility + avoiding ships.

[0039] Scenario 4: Emergency scenarios such as servo motor failure and people falling into the water.

[0040] Data synchronization: EEG signals, operational behavior data, and scene timestamps are synchronized at the millisecond level.

[0041] Post-experiment evaluation: Experts evaluate the participants' performance, including operational procedures and behavioral characteristics.

[0042] This application constructs a target maritime scenario using a ship simulator and correlates it with operational behavior data, ensuring a high degree of correlation between the collected data and high-risk maritime operations. This makes the entire assessment system closely aligned with the actual needs of maritime operations, specifically capturing key operational nodes and environmental stressors that best reflect changes in crew workload, thereby guaranteeing the effectiveness and accuracy of the assessment results in practical applications.

[0043] In some embodiments, when constructing a multi-dimensional feature set, a sliding window method is used to process the power features and the combined indicators to generate a time-series feature matrix. The width of the sliding window is 2 to 10 seconds, and the step size is 0.5 to 2 seconds. The row dimension of the time-series feature matrix is ​​the time point, and the column dimension is the feature parameter.

[0044] Specifically, in the generation of the temporal feature matrix, a 2-second sliding window with a step size of 1 second is used. Each window contains 1024 sampling points, calculated from a 512Hz sampling rate and a 2-second window duration. For each window, feature parameters are extracted, including the power spectral density of five frequency bands (delta, theta, alpha, beta, and gamma), as well as combined features of ratios or weighted features. For each scene's experimental data, after the above sliding window processing, approximately 1800 windows are generated. The feature parameters corresponding to these windows together constitute a feature matrix, providing structured input data for subsequent model training.

[0045] This application constructs a time-series feature matrix using the sliding window technique, transforming static spectral features into dynamic time-series feature sequences. This effectively captures the continuous fluctuations and dynamic evolution of crew workload as the task progresses and the scenario changes, providing a structured data foundation for subsequent model analysis of time-series dependencies.

[0046] In some embodiments, the preprocessing step of the EEG signal specifically includes: bandpass filtering the EEG signal using a finite impulse response filter or an infinite impulse response filter, with a filtering frequency band of 0.1Hz-100Hz, to retain the main cognitive-related frequency bands in the EEG signal while removing DC drift and high-frequency noise; identifying and processing motion artifacts and / or physiological artifacts in the EEG signal using independent principal component analysis combined with manual inspection, and repairing the identified artifact segments using cubic spline interpolation or neighboring data replacement; aligning the EEG signal, operational behavior data, and target nautical scene using scene timestamps to form a time-series dataset, and standardizing the time-series dataset to eliminate signal amplitude deviations caused by individual differences.

[0047] Specifically, the signal preprocessing stage has clearly defined parameters and operating procedures. Data import and format conversion utilizes Matlab R2020b to import .edf format EEG data, converting it to 16-bit integer format, extracting timestamp information, and establishing a correspondence with the behavioral data. Filtering parameters are set to FIR filter: passband 0.5-50Hz, stopband less than 0.1Hz and greater than 60Hz, passband ripple 0.5dB, stopband attenuation 40dB, filter order 128, and zero-phase filtering is performed to avoid phase shift. The artifact processing workflow includes automated detection, manual verification, and repair methods. Automated detection identifies motion artifacts as continuous segments with signal amplitudes greater than ±80 μV or a rate of change greater than 10 μV / sample, and electromyography (EMG) artifacts as signal segments with energy proportions greater than 40% in the 20-50 Hz frequency band. Manual verification involves independently checking the automatically labeled results. Repair methods vary depending on artifact length: short artifacts are repaired using cubic spline interpolation, long artifacts are replaced with the average value of adjacent normal signal segments, and excessively long artifacts are marked as invalid data and excluded from subsequent analysis. For data quality control, data from subjects with an invalid data ratio greater than 10% are removed.

[0048] This application uses operations such as filtering, artifact removal, and data alignment to retain as much effective information related to cognitive activities as possible from the original signal. It transforms the original, noisy multimodal data stream into a clean, regular, and time-synchronized standardized dataset, effectively eliminating individual differences and signal interference, and providing high-quality data input for subsequent accurate feature extraction and model analysis.

[0049] In some embodiments, the step of extracting power features of multiple frequency bands in the preprocessed EEG signal and calculating a combined index related to workload specifically includes: performing spectral analysis on the EEG signal using Fourier transform or wavelet transform, decomposing the EEG signal into five frequency bands: delta, theta, alpha, beta, and gamma, and calculating the power spectral density or average power of each frequency band; and calculating the power ratio or weighted combination value of different frequency bands based on the power spectral density or average power of each frequency band.

[0050] Specifically, in the frequency band power calculation process, a 5-second non-overlapping window is used, with each window containing 2560 sampling points. This is calculated based on a sampling rate of 512Hz and a window duration of 5 seconds. During spectrum analysis, a Hanning window is first added to the signal in each window to reduce spectral leakage and ensure the accuracy of the spectrum analysis. Then, an FFT transform is performed to calculate the signal's power spectral density. Based on this, frequency band integration is performed to obtain the power spectral density values ​​for each frequency band. The integral result for the 0.5Hz-4Hz frequency band is... ; The integral result for the 4Hz-8Hz frequency band, i.e. ; The integral result for the 8Hz-12Hz frequency band, i.e. ; The integral result for the 12Hz-30Hz frequency band, i.e. ; The integral result for the 30Hz-50Hz frequency band, i.e. .

[0051] This application extracts power characteristics of specific frequency bands from electroencephalogram (EEG) signals, transforming complex EEG signals into quantifiable biomarkers that are highly correlated with different cognitive functions such as attention, working memory, and stress arousal. This provides scientifically validated and interpretable neurophysiological indicators for the objective assessment of workload.

[0052] In some embodiments, the formulas for calculating the power ratios of the different frequency bands are: Ratio1 = PSD_θ / PSD_α; where: PSD_θ is the power spectral density of the theta band, and PSD_α is the power spectral density of the alpha band; Ratio2 = PSD_β / (PSD_α + PSD_θ); where: PSD_β is the power spectral density of the beta band, PSD_α is the power spectral density of the alpha band, and PSD_θ is the power spectral density of the theta band; Ratio3 = PSD_γ / (PSD_θ + PSD_α); where: PSD_γ is the power spectral density of the gamma band, PSD_θ is the power spectral density of the theta band, and PSD_α is the power spectral density of the alpha band; Ratio4 = PSD_β / PSD_θ; where: PSD_β is the power spectral density of the beta band, and PSD_θ is the power spectral density of the theta band; the formula for calculating the weighted combination value of the frequency band power is: In the formula: These are the weighting coefficients for each frequency band. Let be the power spectral density of each frequency band, where the weight of each frequency band is determined by principal component analysis.

[0053] This application constructs a composite evaluation index that is more sensitive and stable than single-band power by calculating specific power ratios and weighted combination values. The θ / α ratio reflects the balance between working memory load and cortical relaxation, and its value increases with increased cognitive effort. The γ / (θ + α) ratio quantifies high-frequency neural activity relative to low-frequency oscillations, serving as an indicator of delayed stress response and a key mechanism for maintaining sustained alertness or promoting post-event recovery. The β / θ ratio reflects the dynamic interaction between executive control and working memory, and its changes are closely related to motor response efficiency and reaction time. The β / (α + θ) ratio further captures the dynamics of resource allocation during the decision-making process. These combined indices can more profoundly reveal the interaction and balance between different cognitive systems in the brain, thereby more accurately mapping changes in cognitive resource allocation and stress levels among crew members under complex tasks.

[0054] In some embodiments, the step of constructing a deep learning model comprising a convolutional neural network and a bidirectional long short-term memory network specifically includes: the convolutional neural network extracts local spectral features through a one-dimensional convolutional kernel and reduces the dimensionality through a pooling layer; the bidirectional long short-term memory network captures the temporal dependencies of feature sequences through memory unit state updates; and the model output is a workload level indicating low or high workload.

[0055] Specifically, in the model building phase, the core task is to construct a deep learning model that integrates spectral features and time-series features to achieve accurate workload assessment. The model completes the nonlinear transformation from input features to output workload values ​​through multi-layer function mapping, and its mathematical expression is defined as: ,in, Represents the predicted workload value. For parameterized model functions, This is the set of parameters for the model, and F is the input feature matrix. The model contains three core modules, which respectively implement feature extraction, time series analysis, and result output.

[0056] The feature extraction module uses a convolutional neural network to implement feature mapping, extracting local spectral features from EEG signals through multiple convolutional layers. The formula for the convolutional layer operation is as follows: ,in, This represents the convolution operation. For convolution kernel parameters, For bias terms, For activation function, This represents the output features of the convolutional layer. After the convolution operation, feature dimensionality reduction is performed using a pooling layer. The pooling operation formula is as follows: ,in, The pooling function effectively reduces feature dimensionality while retaining key information. The temporal analysis module employs a bidirectional Long Short-Term Memory (LSTM) network to process the feature sequences output from convolutional layers, capturing the temporal dynamics of the signal. Its memory cell state update rules are implemented through the synergistic action of the input gate, forget gate, and output gate: the input gate controls the inflow of new information, the forget gate determines whether to retain or discard historical states, and the output gate generates hidden states based on the current cell state. Through bidirectional propagation, the LSM network can simultaneously capture past and future temporal dependencies, enhancing its ability to model dynamic signals. The output module outputs workload evaluation results through fully connected layers.

[0057] The model training process optimizes parameters by minimizing the loss function. ,Right now ,in, The loss function is cross-entropy loss for classification tasks and mean squared error loss for regression tasks. The labels are real numbers. In practice, the feature data is divided into training and test sets in a 7:3 ratio. The Adam optimizer is used for parameter updates with a learning rate of 0.001. An early stopping strategy is employed: training stops when the validation set loss shows no decrease for five consecutive epochs to prevent overfitting. Model evaluation is performed using metrics such as accuracy, area under the ROC curve, precision, and recall to ensure good generalization ability across different application scenarios.

[0058] This application employs a model architecture combining convolutional neural networks and bidirectional long short-term memory networks. It leverages the advantages of convolutional neural networks in extracting local spatial spectral features and bidirectional long short-term memory networks in modeling long temporal dependencies. This enables the model to capture subtle patterns in EEG signals and understand the temporal context of workload states, thereby achieving accurate and robust classification of workloads.

[0059] In some embodiments, the steps of training, optimizing, and evaluating the deep learning model specifically include: training the deep learning model by optimizing parameters through minimizing the loss function; optimizing the deep learning model using an early stopping strategy, stopping training when the validation set loss does not decrease for 5 consecutive epochs; and evaluating the deep learning model using metrics such as accuracy, area under the ROC curve, precision, and recall, combined with cross-validation, case validation, and axiomatic validation.

[0060] Specifically, to clarify the contribution of each neurophysiological feature to the workload assessment results, the Shapley Additive Explanations (SHAP) method was used to quantify the feature influence. By calculating the average absolute SHAP value of all features in the model decision, the core predictive factors were identified: the theta band had an average absolute SHAP value of 0.4346, indicating that the theta band is the primary influencing factor. Its power change mainly reflects fluctuations in working memory load. When theta power increases, the model's prediction probability of a high-load state significantly increases, consistent with the neural mechanism of increased working memory resource consumption. The alpha band had an average absolute SHAP value of 0.3609, indicating that the alpha band is a secondary influencing factor. Its power suppression mainly reflects attention level. When alpha power decreases, the model is more inclined to judge a high-load state, consistent with the neural activity patterns during periods of high concentration. The average absolute SHAP values ​​of other features, such as the beta and gamma bands and combined indicators, were all below 0.2, indicating a weaker impact on the model output, further validating the effectiveness of theta and alpha bands as core biomarkers of workload.

[0061] In maritime scenarios, fluctuations in workload indices can be predicted through dynamic changes in EEG signal frequency band characteristics and combined indicators. Studies have shown that the theta / alpha ratio, as a core working memory workload indicator, can increase by approximately 70% in emergency situations. Its fluctuation amplitude is negatively correlated with the crew's familiarity with operational procedures. When this ratio increases by more than 30% from baseline within 10 seconds, it indicates a possible decision-making delay. The gamma / (theta+alpha) ratio, as a stress arousal marker, increases with a delay in scenarios such as rudder malfunctions. Its peak time usually lags behind operational errors by about 40-60 seconds and can be used to predict stress compensation responses after errors. The beta / (alpha+theta) ratio is closely related to cognitive-motor coordination. When it increases by approximately 40% in collision scenarios, it is accompanied by operations that violate international collision avoidance rules. When this ratio increases by more than 20% within three consecutive sliding windows, combined with behavioral data showing a rudder angle adjustment frequency of more than 5 times / minute, it can predict the risk of illegal steering. By learning the temporal characteristics of these indices through the CNN-BiLSTM model, the dynamic correlation between "index fluctuations and behavioral responses" can be effectively captured, enabling early prediction of high workload states and providing accurate basis for real-time intervention.

[0062] When the power of the theta band increases monotonically, the high load prediction probability output by the model also increases monotonically, which verifies the positive correlation between neurophysiological characteristics and workload, and is consistent with the neuroscience theory that increased working memory load is accompanied by increased theta power. When any two or more features act together, the change in the predicted probability of high-load conditions is greater than the sum of the changes in the individual effects of each feature. This verifies the risk accumulation characteristics of the multi-factor synergistic effect and is consistent with the superposition law of cognitive load in complex work scenarios.

[0063] Emergency scenarios including ship collision avoidance, steering gear malfunction handling, and fog navigation were selected, and the model was validated using independent test data from multiple subjects. The results show that the model's prediction accuracy is above 96% in all navigation scenarios, with an area under the ROC curve of 0.99. Specifically, the recall rate for high-load conditions is 97%, and the precision rate is 96%, indicating that the model can stably capture workload changes under different emergency scenarios and meets the accuracy requirements of practical applications.

[0064] This application employs the aforementioned model training, optimization, and evaluation strategies to ensure that the constructed deep learning model can fully learn the patterns in the training data while maintaining good generalization ability and avoiding overfitting. This ensures that the model can maintain stable and reliable evaluation performance even under unknown data or new scenarios.

[0065] In some embodiments, the triggering conditions for the graded warning include: the high load probability output by the deep learning model lasts for 10 seconds, and the Ratio1 ratio increases by 30%~50% from the baseline or the Ratio2 ratio increases for 10 seconds, triggering a level one warning; the high load probability output by the deep learning model lasts for 30 seconds, and the Ratio3 ratio increases by ≥30% from the baseline or the Ratio4 ratio increases for ≥30%, and the collision avoidance reaction time is at the scene average level, triggering a level two warning.

[0066] Specifically, the load early warning module constructs a multi-level early warning mechanism based on the correlation between workload index fluctuations and crew operational behavior, enabling real-time intervention from abnormal neurophysiological signals to unsafe behavioral risks. This module triggers targeted early warnings by combining the high load probability values ​​output by a pre-trained CNN-BiLSTM model with the dynamic characteristics of the workload index and ship handling behavior indicators, as detailed below: A Level 1 warning indicates high cognitive load, triggered when any of the following conditions are met: The high load probability output by the model lasts for 10 seconds, and the theta / alpha ratio increases by 30%~50% from the baseline or the beta / (alpha+theta) ratio increases for 10 seconds.

[0067] Level 2 warning indicates high load accompanied by operational risk, and is triggered when any of the following conditions are met: The model outputs a high load probability for 30 seconds, and the gamma / (theta+alpha) ratio increases by ≥30% compared to the baseline, the theta / alpha ratio continues to increase, and the collision avoidance reaction time is higher than the scene average.

[0068] This application constructs a forward-looking risk warning mechanism based on physiological evidence by combining the multi-level early warning triggering conditions with the combined indicators of model output probability and EEG power ratio. This mechanism can not only identify the current high-load state, but also predict the potential risk of operational performance decline, thereby achieving graded and proactive safety intervention.

[0069] Please refer to Figure 3 On the other hand, the present invention also provides a deep learning-based crew workload assessment system, comprising: a data acquisition module for acquiring crew members' EEG signals and operational behavior data in a target maritime scenario, and preprocessing the EEG signals; a feature set construction module for extracting power features of multiple frequency bands from the preprocessed EEG signals, calculating combined indicators related to workload, and constructing a multi-dimensional feature set by combining the target maritime scenario and corresponding operational behavior data; a model construction module for constructing a deep learning model including a convolutional neural network and a bidirectional long short-term memory network, using the multi-dimensional feature set as input and the workload level as output, and training, optimizing, and evaluating the deep learning model; and an early warning module for inputting the real-time acquired and preprocessed crew members' EEG signals into the trained deep learning model, outputting the current workload level, and triggering graded early warnings based on the current workload level and the operational behavior corresponding to the current maritime scenario.

[0070] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0072] The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention's specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A deep learning-based method for assessing crew workload, characterized by the following steps: include: Collect the electroencephalogram (EEG) signals and operational behavior data of crew members in the target maritime scenario, and preprocess the EEG signals; Power features of multiple frequency bands were extracted from the preprocessed EEG signals, and combined indicators related to workload were calculated. A multi-dimensional feature set was constructed by combining the target maritime scenario and the corresponding operational behavior data. A deep learning model comprising a convolutional neural network and a bidirectional long short-term memory network is constructed. The multi-dimensional feature set is used as input, and the workload level is used as output. The deep learning model is then trained, optimized, and evaluated. The real-time collected and preprocessed EEG signals of the crew are input into the trained deep learning model, which outputs the current workload level and triggers graded warnings based on the operational behavior corresponding to the current workload level and the current navigation scenario.

2. The deep learning-based crew workload assessment method according to claim 1, characterized in that, The target navigation scenario is constructed using a ship simulator and includes at least one of the following: open water cruising, narrow channel meeting, low visibility fog navigation avoidance, emergency response to personnel falling into the water, and emergency handling of steering gear failure. The operational behavior data includes whether to adjust the rudder angle, accelerate or decelerate the speed, and / or whether to take collision avoidance actions. The multi-dimensional feature set includes EEG signals, target navigation scenarios, operational behavior data, and combined indicators related to workload.

3. The deep learning-based crew workload assessment method according to claim 2, characterized in that, When constructing a multi-dimensional feature set, the power feature and the combined index are processed using a sliding window method to generate a time-series feature matrix. The width of the sliding window is 2 to 10 seconds, and the step size is 0.5 to 2 seconds. The row dimension of the time-series feature matrix is ​​the time point, and the column dimension is the feature parameter.

4. The deep learning-based crew workload assessment method according to claim 1, characterized in that, The preprocessing steps for the electroencephalogram (EEG) signals specifically include: The EEG signal is bandpass filtered using a finite impulse response filter or an infinite impulse response filter, with a filtering frequency range of 0.1Hz-100Hz, in order to retain the main cognitive-related frequency bands in the EEG signal while removing DC drift and high-frequency noise. The motion artifacts and / or physiological artifacts in the EEG signals are identified and processed by independent principal component analysis combined with manual inspection. The identified artifact segments are repaired by cubic spline interpolation or neighboring data replacement. The EEG signals, operational behavior data, and target navigation scene are aligned using scene timestamps to form a time-series dataset. The time-series dataset is then standardized to eliminate signal amplitude deviations caused by individual differences.

5. The deep learning-based crew workload assessment method according to claim 1, characterized in that, The steps of extracting power characteristics of multiple frequency bands from the preprocessed EEG signal and calculating combined indicators related to workload specifically include: Fourier transform or wavelet transform is used to perform spectral analysis on the EEG signal, decomposing the EEG signal into five frequency bands: delta, theta, alpha, beta, and gamma, and calculating the power spectral density or average power of each frequency band. Calculate the power ratio or weighted combination value of different frequency bands based on the power spectral density or average power of each frequency band.

6. The deep learning-based crew workload assessment method according to claim 5, characterized in that, The formula for calculating the power ratio of the different frequency bands is as follows: Ratio1 = PSD_θ / PSD_α; where: PSD_θ is the power spectral density of the theta band, and PSD_α is the power spectral density of the alpha band; Ratio2 = PSD_β / (PSD_α + PSD_θ); where: PSD_β is the power spectral density of the beta band, PSD_α is the power spectral density of the alpha band, and PSD_θ is the power spectral density of the theta band. Ratio3 = PSD_γ / (PSD_θ + PSD_α); where: PSD_γ is the power spectral density of the gamma band, PSD_θ is the power spectral density of the theta band, and PSD_α is the power spectral density of the alpha band. Ratio4 = PSD_β / PSD_θ; where: PSD_β is the power spectral density of the beta band, and PSD_θ is the power spectral density of the theta band; The formula for calculating the power weighted combination value of the frequency band is: ; In the formula: These are the weighting coefficients for each frequency band. Let be the power spectral density of each frequency band, where the weight of each frequency band is determined by principal component analysis.

7. The deep learning-based crew workload assessment method according to claim 1, characterized in that, The steps for constructing a deep learning model that includes a convolutional neural network and a bidirectional long short-term memory network specifically include: The convolutional neural network extracts local spectral features through one-dimensional convolutional kernels and reduces the dimensionality through pooling layers; the bidirectional long short-term memory network captures the temporal dependencies of feature sequences through memory unit state updates. The model output is the load level for low or high operating load.

8. The deep learning-based crew workload assessment method according to claim 1, characterized in that, The steps of training, optimizing, and evaluating the deep learning model specifically include: The deep learning model is trained by optimizing the parameters by minimizing the loss function; The deep learning algorithm is optimized using an early stopping strategy, where training is stopped when the validation set loss does not decrease for 5 consecutive epochs. The deep learning model is evaluated using metrics such as accuracy, area under the ROC curve, precision, and recall, combined with cross-validation, case validation, and axiomatic validation.

9. The deep learning-based crew workload assessment method according to claim 6, characterized in that, The triggering conditions for the tiered early warning include: If the high load probability output by the deep learning model lasts for 10 seconds, and the Ratio1 ratio increases by 30% to 50% from the baseline or the Ratio2 ratio increases for 10 seconds, a level one warning is triggered. If the high load probability output by the deep learning model lasts for 30 seconds, and the Ratio3 ratio increases by ≥30% compared to the baseline or the Ratio4 ratio continues to increase, and the collision avoidance reaction time is at the scene average level, a level 2 warning is triggered.

10. A deep learning-based system for assessing crew workload, characterized in that, include: The data acquisition module is used to collect the electroencephalogram (EEG) signals and operational behavior data of crew members in the target maritime scenario, and to preprocess the EEG signals. The feature set construction module is used to extract power features of multiple frequency bands in the preprocessed EEG signal and calculate the combined indicators related to workload. It combines the target maritime scenario and the corresponding operational behavior data to construct a multi-dimensional feature set. The model building module is used to build a deep learning model that includes a convolutional neural network and a bidirectional long short-term memory network. It takes the multi-dimensional feature set as input and the workload level as output to train, optimize and evaluate the deep learning model. The early warning module is used to input the real-time collected and preprocessed EEG signals of crew members into a trained deep learning model, output the current workload level, and trigger graded early warnings based on the current workload level and the corresponding operational behavior in the current navigation scenario.

Images