A crew workload assessment method and system based on multi-source data fusion

Abstract

The application discloses a kind of crew workload evaluation method and system based on multi-source data fusion, comprising: step 1, multi-source data acquisition is carried out, the multi-source data includes objective data, sensory monitoring data;Step 2, the feature extraction of the multi-source data is carried out;Step 3, based on physiological data and sensory monitoring data, through feature layer fusion and decision layer fusion;Step 4, the real-time workload level of crew is presented by visualizing the workload evaluation result, and when workload index exceeds preset threshold, trigger early warning prompt, remind crew and manager to pay attention to fatigue risk.The application realizes the comprehensive accurate evaluation and real-time visual presentation of crew load state by comprehensively utilizing multi-source heterogeneous data, and constructs effective early warning mechanism, has important application value in improving crew work safety and operating efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a method and system for assessing crew workload based on multi-source data fusion. Background Technology

[0002] With the increasing size, intelligence, and complexity of modern ships, the workload and psychological stress on crew members are rising. Prolonged periods of high-intensity work can easily lead to fatigue and loss of concentration among crew members, resulting in operational errors, misjudgments, and other problems that seriously threaten the safety of ship navigation and operations. Therefore, timely, accurate, and comprehensive assessment of crew members' workload levels is of great significance for accident prevention, protecting crew health, and improving work efficiency.

[0003] Traditional methods for assessing crew workload primarily rely on crew members' subjective self-assessment scales, such as the Fatigue Self-Rating Scale and the NASA-TLX Task Load Index. While these methods are simple to operate, they are easily influenced by crew members' subjective opinions, making it difficult to guarantee the objectivity and reliability of the assessment results. In recent years, with the development of emerging technologies such as the Internet of Things and artificial intelligence, using smart wearable devices to collect crew members' physiological data and analyzing their workload status through machine learning algorithms has become a new technological trend.

[0004] This invention is proposed against this background, aiming to overcome the limitations of traditional methods and provide a method for assessing crew workload based on multi-source data fusion, so as to more accurately, comprehensively and timely grasp the crew's workload level, effectively prevent safety hazards caused by overwork, and protect the health of crew members and the safety of ship operations. Summary of the Invention

[0005] To address the shortcomings of the existing technology, this invention provides a method for assessing crew workload based on multi-source data fusion, the method comprising:

[0006] Step 1: Collect multi-source data, including objective data and sensory monitoring data.

[0007] Step 2: Extract features from the multi-source data;

[0008] Step 3: Based on physiological data and sensory monitoring data, feature layer fusion and decision layer fusion are performed;

[0009] Step 4: Visualize the workload assessment results to show the crew's real-time workload level. When the workload index exceeds the preset threshold, trigger an early warning to remind crew members and managers to pay attention to the risk of fatigue.

[0010] The objective data collection includes using portable physiological monitoring equipment to collect real-time physiological data such as the crew's heart rate and electromyography.

[0011] The sensory monitoring data includes video and audio sensors deployed in the work area to record crew members' facial expressions and voice data.

[0012] Among them, fatigue monitoring data collection is achieved through cross-sensory coordination via visual and auditory channels.

[0013] The visual channel includes capturing facial images and eye movement data of crew members via cameras for facial expression recognition and eye-tracking analysis.

[0014] The auditory channel includes collecting voice data from crew members via microphones for voice emotion calculation and fatigue assessment.

[0015] The extraction of physiological data features from crew members includes at least the following:

[0016] Analyze heart rate data to extract features such as mean heart rate and heart rate variability.

[0017] Time-frequency analysis was performed on electromyography data to extract features related to muscle fatigue.

[0018] For the visual channel, computer vision algorithms are used to extract facial expression features and eye movement features, including eye closure.

[0019] Furthermore, eye-tracking pattern analysis was performed on the eye-tracking data to extract features such as fixation time and blink frequency.

[0020] For the auditory channel, a speech emotion computing model is used to extract speech emotion features, which include volume, pitch, and speech rate.

[0021] The feature layer fusion process employs adaptive feature weighting. In this adaptive feature weighting, adaptive feature weighting coefficients are determined for different data sources, such as physiological data and sensory monitoring data. The weighting coefficients for each data source are dynamically adjusted by analyzing its signal-to-noise ratio, data quality, and correlation with workload.

[0022] Among them, the decision-making level fusion is based on evidence theory, which regards physiological data as direct evidence and sensory monitoring data as indirect evidence. By combining the evidence, the probability distribution of the overall workload status is obtained.

[0023] Among them, a preliminary workload assessment is conducted based on the feature vector of direct evidence.

[0024] Construct a Bayesian network model connecting workload influencing factors and workload status.

[0025] Bayesian inference is performed using feature vectors of indirect evidence.

[0026] The workload classifier or regression model is trained using a machine learning model, with [HR_mean,SDNN,RMSSD] and [RMS,MPF] as input features, and outputs workload levels W_1 and W_2 respectively.

[0027] in,

[0028] HR_mean, SDNN, and RMSSD are used to extract the mean heart rate, SDNN, and RMSSD from the heart rate data, respectively.

[0029] RMS and MPF are the root mean square value and average power frequency extracted from electromyography data, respectively.

[0030] W_initial=W_HR_norm*W_1+W_EMG_norm*W_2,

[0031] Where W_HR_norm and W_EMG_norm are the normalized quality weights of heart rate data and electromyography data obtained in step 2, respectively.

[0032] W_1 and W_2 are the workload levels determined based on heart rate and electromyography data, respectively.

[0033] The weighted result is denoted as the initial workload assessment result W_initial, and used as the prior probability P(W) in subsequent Bayesian inference.

[0034] A Bayesian network model is constructed, where nodes represent workload states and indirect evidence types, and edges represent the dependencies between them.

[0035] Assign a conditional probability table CPT to each node, representing the probability that the node takes a different value when its parent node takes a different value.

[0036] Indirect evidence features are used as observation variables, i.e. known conditions, and the values ​​of the corresponding child nodes are fixed in the Bayesian network.

[0037] For each indirect evidence feature node, the posterior probability distribution of its relation to the workload state is calculated using a Bayesian inference algorithm.

[0038] We take a weighted average of the features corresponding to the visual channel data and the speech channel data respectively to obtain the posterior probability distribution of the two channels.

[0039] The posterior probability distributions of the two channels are weighted and combined to obtain the final posterior probability distribution.

[0040] Finally, based on the posterior probability distribution obtained by weighted combination, the workload state with the highest probability is selected as the final evaluation result.

[0041] The workload states include: low load, medium load, or high load.

[0042] The present invention also discloses a crew workload assessment system based on multi-source data fusion. The system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program in the memory to perform the aforementioned method.

[0043] This invention proposes a multi-source data fusion-based method for assessing crew workload. By comprehensively utilizing heterogeneous data from multiple sources, it achieves a comprehensive and accurate assessment of crew workload status and real-time visualization, and establishes an effective early warning mechanism. This method has significant application value in improving crew work safety and operational efficiency. This invention effectively solves the technical bottlenecks of traditional workload assessment methods and demonstrates the innovative application of multi-source data fusion technology in the field of intelligent shipping. Attached Figure Description

[0044] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0045] Figure 1 This is a flowchart illustrating a method for assessing crew workload based on multi-source data fusion according to an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0047] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0048] It should be understood that although the terms first, second, third, etc., may be used to describe... in the embodiments of the present invention, these... should not be limited to these terms. These terms are only used to distinguish... For example, first... may also be referred to as second... without departing from the scope of the embodiments of the present invention, and similarly, second... may also be referred to as first...

[0049] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0050] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0051] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0052] This invention proposes a method for assessing crew workload based on multi-source data fusion. This method comprehensively utilizes physiological data, cross-sensory collaborative immersive fatigue monitoring data, and multi-perspective workload assessment data. Through multi-level data fusion, it achieves a comprehensive and accurate assessment of crew workload.

[0053] This invention comprehensively utilizes multi-source data, including objective data and sensory monitoring data, to overcome the limitations of single data sources. It assesses crew workload levels from multiple dimensions, including physiological, behavioral, and environmental factors, improving the accuracy and reliability of the assessment results. Employing a combination of feature-level fusion and decision-level fusion, it addresses the complementarity and redundancy of multi-source data at both the data and decision levels, leveraging the advantages of different data sources to extract more comprehensive and refined workload characteristics, resulting in more robust assessment results. The workload assessment results are presented in real-time in a visual manner, intuitively reflecting crew fatigue levels, facilitating crew self-regulation and real-time monitoring by managers, effectively preventing safety accidents caused by fatigued driving. A warning threshold for the workload index is set; when a crew member's workload level exceeds the threshold, an automatic warning is triggered, reminding the crew to rest through sound and light, and notifying managers to take necessary intervention measures, effectively improving the safety of ship operations.

[0054] like Figure 1 As shown, the present invention proposes a method for assessing crew workload based on multi-source data fusion, comprising:

[0055] Step 1: Collect multi-source data, including objective data and sensory monitoring data.

[0056] Objective data collection includes using portable physiological monitoring devices to collect real-time physiological data such as the crew's heart rate and electromyography.

[0057] The sensory monitoring data requires the deployment of video and audio sensors in key or major operating areas of the ship to record crew members' facial expressions and voice data. Specifically, fatigue monitoring data collection is achieved through cross-sensory collaboration via visual and auditory channels.

[0058] Visual channel: Captures crew members' facial images and eye movement data via cameras for facial expression recognition and eye-tracking analysis.

[0059] Auditory channel: Collects voice data from crew members through microphones for voice emotion calculation and fatigue assessment.

[0060] This invention assesses crew workload based on multi-source data fusion. Specifically, it uses data from crew-worn physiological monitoring devices to infer crew workload using intelligent algorithms. Furthermore, it uses collected video and audio sensor data to demonstrate the intensity of the crew's workload.

[0061] Step 2: Extract features from the multi-source data.

[0062] Before feature extraction, preprocessing operations such as timestamp alignment, data cleaning, and missing value imputation are performed on the collected multi-source heterogeneous data to improve data quality and consistency.

[0063] The extraction of physiological data features from crew members includes at least the following:

[0064] Analyze heart rate data to extract features such as mean heart rate and heart rate variability.

[0065] Time-frequency analysis was performed on electromyography data to extract features related to muscle fatigue.

[0066] Among these methods, heart rate data is analyzed to extract features of mean heart rate and heart rate variability.

[0067] Mean heart rate represents the average heart rate over a period of time, reflecting the overall heart rate status of the crew.

[0068] HR_mean = Σ(HR_i) / N,

[0069] Where HR_i is the i-th heart rate data point, and N is the total number of heart rate data points.

[0070] Heart rate variability measures the degree of change in heart rate over a period of time, reflecting a crew member's ability to regulate heart rate. The heart rate variability indicators include SDNN (standard deviation of all RR intervals) and RMSSD (root mean square of the difference between adjacent RR intervals).

[0071] SDNN=sqrt(Σ((RR_i-RR_mean)^2) / (N-1)),

[0072] RMSSD=sqrt(Σ((RR_i+1-RR_i)^2) / (N-1)),

[0073] Where RR_i is the i-th RR interval (the time interval between adjacent heartbeats), RR_mean is the average value of the RR intervals, and N is the total number of RR intervals.

[0074] Among these methods, time-frequency analysis of electromyography data was performed to extract features related to muscle fatigue levels.

[0075] The root mean square (RMS) value reflects the energy level of electromyographic signals and can be used to assess muscle activity intensity and fatigue level.

[0076] RMS = sqrt(Σ(EMG_i^2) / N),

[0077] Where EMG_i is the i-th electromyography data point, and N is the total number of electromyography data points.

[0078] The average power frequency (MPF) is the centroid frequency of the power spectrum of electromyographic signals and can also reflect the spectral changes caused by muscle fatigue.

[0079] MPF = Σ(f_i*P_i) / Σ(P_i)

[0080] Where f_i is the center frequency of the i-th frequency bin, and P_i is the power spectral density value of the i-th frequency bin.

[0081] The feature extraction of the crew's sensory monitoring data includes:

[0082] For the visual channel, computer vision algorithms are used to extract facial expression features and eye movement features, such as eye closure and gaze direction. Eye movement pattern analysis is performed on the eye movement data to extract features such as fixation time and blink frequency.

[0083] For the auditory channel, a speech emotion computing model is used to extract speech emotion features, such as volume, pitch, and speech rate.

[0084] For the visual channel, computer vision algorithms are used to extract facial expression features and eye movement features.

[0085] Facial landmark detection algorithms are used to detect facial landmarks, and then facial expression features are calculated based on the position and movement of the landmarks.

[0086] Based on the detected facial key points, facial expression features are calculated.

[0087] Facial expressions can be quantified by calculating geometric features such as the distance, angle, and area between key points. For example, by calculating the distance between key points on the upper and lower eyelids, the degree of eye opening and closing can be estimated; by calculating changes in the position of the corners of the mouth, expressions such as smiling or surprise can be determined.

[0088] Among these methods, eye movement pattern analysis is performed on the eye movement data to extract features such as fixation time and blink frequency.

[0089] Eye movement pattern analysis utilizes eye keypoint data from facial keypoints to perform eye movement pattern analysis. This includes: extracting the coordinates of keypoints for the left and right eyes, calculating the center position of the eyes, tracking changes in the eye center position to obtain eye movement trajectories, and calculating features such as fixation time and blink frequency based on the eye movement trajectories.

[0090] Among them, the average gaze duration (Gaze_duration_mean) represents the average duration of a crew member's gaze within the corresponding region of interest.

[0091] Gaze_duration_mean=Σ(Gaze_duration_i) / N

[0092] Where Gaze_duration_i is the duration of the i-th gaze, and N is the total number of gazes.

[0093] Blink frequency represents the number of blinks per unit of time, reflecting the crew's fatigue level and attention level.

[0094] Blink_frequency=N_blink / T,

[0095] Where N_blink is the total number of blinks and T is the total measurement time.

[0096] Facial features include eye closure, which is determined by calculating the ratio of the distance between the upper and lower eyelids to the width of the eye.

[0097] Eye_closure=(P_upper-P_lower) / Eye_width,

[0098] Where P_upper and P_lower are the vertical coordinates of the key points of the upper and lower eyelids, respectively, and Eye_width is the width of the eye.

[0099] For the auditory channel, a speech emotion computing model is used to extract speech emotion features.

[0100] First, the recorded audio data is preprocessed, such as removing silent segments and reducing noise, to improve audio quality.

[0101] Then, a speech emotion recognition model, such as a speech feature extraction model based on algorithms like SVM and DNN, is used to extract the emotional features of the speech. These emotional features specifically include:

[0102] 1. Volume:

[0103] Calculate the root mean square (RMS) value of the speech signal, which represents the loudness of the speech.

[0104] Volume = sqrt(Σ(x_i^2) / N),

[0105] Where x_i is the i-th sampling point of the speech signal, and N is the total number of sampling points.

[0106] 2. Pitch:

[0107] Calculate the fundamental frequency (F0) of the speech signal, which represents the pitch of the speech.

[0108] The fundamental frequency can be estimated using time-domain based algorithms (such as the YIN algorithm) or frequency-domain based algorithms (such as the Cepstrum algorithm).

[0109] 3. Speaking speed:

[0110] Calculate the number of syllables or words in a speech unit of time to indicate the speed of speech.

[0111] Speech_rate=N_syllables / T,

[0112] Where N_syllables is the total number of syllables, and T is the total duration of the speech.

[0113] Step 3: Based on physiological data and sensory monitoring data, feature layer fusion and decision layer fusion are performed.

[0114] In this process, feature layer fusion performs adaptive feature weighting.

[0115] Among them, the decision-making level fusion is based on evidence theory, which regards physiological data as direct evidence and sensory monitoring data as indirect evidence. By combining the evidence, the probability distribution of the overall workload status is obtained.

[0116] In the adaptive feature weighting process, adaptive feature weighting coefficients are determined for different data sources, such as physiological data and sensory monitoring data. The weighting coefficients for each data source are dynamically adjusted by analyzing its signal-to-noise ratio, data quality, and correlation with workload.

[0117] Physiological data quality assessment is the process of analyzing the quality of collected heart rate, electromyography, and eye movement data. By calculating indicators such as signal integrity, signal-to-noise ratio, data validity, and calibration accuracy, the quality of each type of physiological data source can be quantified.

[0118] For heart rate and electromyography (EMG) data, signal integrity measures the amount of missing data during the data acquisition process. The degree of signal integrity can be assessed by calculating the proportion of missing data points to the total number of data points.

[0119] The missing data rate is calculated as: missing data points / total data points. Missing data points represent the number of data points that were not successfully collected, and total data points represent the total number of data points collected.

[0120] Furthermore, for heart rate and electromyography (EMG) data, the signal-to-noise ratio (SNR) measures the proportion of useful information to noise in the signal. The signal quality can be assessed by calculating the ratio of signal power to noise power.

[0121] SNR = 10 * log10 (signal power / noise power)

[0122] Signal power = Σ(x_i^2) / N, where x_i is the signal amplitude and N is the signal length;

[0123] Noise power = Σ(n_i^2) / N, where n_i is the noise amplitude and N is the noise length;

[0124] Signal power represents the average energy of a signal, while noise power represents the average energy of noise.

[0125] Among them, sensory monitoring data quality assessment is the process of analyzing the quality of collected facial expressions, eye movements (visual channel), and speech data.

[0126] Image sharpness measures the clarity of a facial expression image. It is assessed by calculating the sum of the absolute values ​​of the gradients between adjacent pixels.

[0127] Clarity=Σ(abs(I(x,y)-I(x+1,y))+abs(I(x,y)-I(x,y+1))) / (2*W*H),

[0128] Where I(x,y) is the pixel value of the image at position (x,y), and W and H are the width and height of the image, respectively.

[0129] Face detection rate measures the proportion of successfully detected faces in facial expression data. The effectiveness of face detection can be evaluated by calculating the percentage of frames in which faces were successfully detected out of the total number of frames.

[0130] Detection rate = Number of frames that successfully detected a face / Total number of frames.

[0131] Among them, the number of frames in which a face was successfully detected represents the number of frames in which the algorithm successfully identified a face, and the total number of frames represents the total number of frames of facial expression data.

[0132] Furthermore, for eye-tracking data (visual channel), the pupil detection rate measures the proportion of frames in the eye-tracking data where pupils are successfully detected. The effectiveness of pupil detection can be evaluated by calculating the percentage of frames in which pupils are successfully detected out of the total number of frames.

[0133] Detection rate = Number of frames that successfully detected the pupil / Total number of frames

[0134] Among them, the number of frames in which the pupil was successfully detected represents the number of frames in which the algorithm successfully identified the pupil, and the total number of frames represents the total number of frames in the eye-tracking data.

[0135] For features extracted from speech data, the signal-to-noise ratio (SNR) measures the proportion of speech to noise in the speech signal. By calculating the ratio of speech signal power to noise power, the quality of the speech signal can be evaluated.

[0136] SNR = 10 * log10 (speech signal power / noise power);

[0137] Speech signal power = Σ(s_i^2) / N, where s_i is the amplitude of the speech signal and N is the length of the speech signal;

[0138] Noise power = Σ(n_i^2) / N, where n_i is the noise amplitude and N is the noise length;

[0139] Speech signal power represents the average energy of the speech signal, while noise power represents the average energy of the noise.

[0140] Next, in order to comprehensively evaluate the quality of each data source, it is necessary to convert each quality indicator into a unified quality score. By normalizing the quality indicators, indicators with different dimensions can be mapped to the same scale. Min-max normalization scales the range of quality indicator values ​​to the [0,1] interval.

[0141] To dynamically adjust the weights of each data source, calculations need to be performed by combining the data quality score and the initial weights. The final weights for each data source can be obtained through quality weight updates and weight normalization.

[0142] The quality weight update multiplies the data quality score by the initial weight to obtain the weight that takes quality factors into account.

[0143] W_i = Q_i * W_i_init;

[0144] Where W_i is the quality weight of the i-th data source, Q_i is the quality score of the i-th data source, and W_i_init is the initial weight of the i-th data source.

[0145] The quality scores of each data source are obtained in the following ways:

[0146] Heart rate data quality score Q_HR:

[0147] Q_HR = (1 - Missing Rate_HR) * α_HR + SNR_HR * β_HR,

[0148] Wherein, the missing rate _HR and SNR_HR are the missing rate and signal-to-noise ratio of the heart rate data, respectively, and α_HR and β_HR are the weighting coefficients of the missing rate and signal-to-noise ratio, satisfying α_HR+β_HR=1.

[0149] Electromyography (EMG) data quality score Q_EMG:

[0150] Q_EMG=(1-missing rate_EMG)*α_EMG+SNR_EMG*β_EMG,

[0151] Wherein, the missing rate_EMG and SNR_EMG are the missing rate and signal-to-noise ratio of the electromyography data, respectively, and α_EMG and β_EMG are the weighting coefficients of the missing rate and signal-to-noise ratio, satisfying α_EMG+β_EMG=1.

[0152] Visual channel data quality score Q_Visual:

[0153] Q_Visual = Clarity_Visual * λ_Visual + Face Detection Rate_Visual * μ_Visual + Pupil Detection Rate_Visual * ν_Visual

[0154] Where, sharpness_Visual, face detection rate_Visual, and pupil detection rate_Visual are the image sharpness, face detection rate, and pupil detection rate of the visual channel data, respectively, and λ_Visual, μ_Visual, and ν_Visual are the weight coefficients of the three indicators, satisfying λ_Visual + μ_Visual + ν_Visual = 1.

[0155] Auditory channel data quality score Q_Audio:

[0156] Q_Audio = SNR_Audio

[0157] Where SNR_Audio is the signal-to-noise ratio of the speech data.

[0158] Furthermore, the weights corresponding to the direct evidence features and indirect evidence features are normalized by dividing the quality weight by the sum of all weights, ensuring that the sum of the weights of the direct evidence features and the sum of the weights of the indirect evidence features are both 1.

[0159] W_HR_norm and W_EMG_norm are the normalized quality weights for heart rate and electromyography data within the direct evidence feature group, respectively.

[0160] W_visual_norm and W_Audio_norm are the normalized quality weights for the visual and audio channel data within the indirect evidence feature group, respectively.

[0161] The aforementioned decision-level fusion includes reasoning about workload based on evidence theory.

[0162] Physiological data is considered direct evidence, providing direct information about workload status. Sensory monitoring data is considered indirect evidence, providing indirect information about factors influencing workload.

[0163] Direct evidence feature vectors and indirect evidence feature vectors are used as evidence within the evidence theory framework, representing the degree of support for workload status from physiological data and sensory monitoring data, respectively. Specifically, this includes:

[0164] Step S1: Conduct a preliminary workload assessment based on the feature vector of direct evidence.

[0165] Use machine learning models (such as support vector machines, SVM) to train workload classifiers or regression models, with [HR_mean, SDNN, RMSSD] and [RMS, MPF] as input features, and output workload levels W_1 and W_2 respectively.

[0166] in,

[0167] HR_mean, SDNN, and RMSSD are the features extracted from the heart rate data, namely the mean heart rate, SDNN, and RMSSD, respectively.

[0168] RMS and MPF are the features extracted from electromyography data: root mean square value and average power frequency, respectively.

[0169] W_initial=W_HR_norm*W_1+W_EMG_norm*W_2,

[0170] Where W_HR_norm and W_EMG_norm are the normalized quality weights of heart rate data and electromyography data obtained in step 2, respectively.

[0171] W_1 and W_2 are the workload levels determined based on heart rate and electromyography data, respectively.

[0172] The weighted result is denoted as the initial workload assessment result W_initial, and used as the prior probability P(W) in subsequent Bayesian inference.

[0173] Step S2: Construct a Bayesian network model between workload influencing factors and workload status.

[0174] Based on domain knowledge and data analysis, the causal relationships and conditional dependencies between indirect evidence features and workload states are determined. A Bayesian network model is constructed, where nodes represent workload states and indirect evidence types, and edges represent dependencies between them. For example, in the Bayesian network model, the workload state node is the parent node, and each indirect evidence feature is a child node.

[0175] Each node is assigned a conditional probability table (CPT), representing the probability that the node will take a different value given different values ​​of its parent node. The CPT can be obtained through expert knowledge or learned from data.

[0176] Based on domain knowledge and data analysis, causal and conditional dependencies between indirect evidence features and workload status are determined. For example, it can be hypothesized that blink frequency and average fixation time are both influenced by workload status, and that, given workload status, blink frequency and average fixation time are conditionally independent. These hypotheses can be verified based on expert knowledge or through data analysis.

[0177] Construct a Bayesian network model where nodes represent workload states and indirect evidence types, and edges represent dependencies between them. For example, in a Bayesian network model, the workload state node is the parent node, and the blink frequency node and the average fixation time node are child nodes.

[0178] A conditional probability table (CPT) is assigned to each node, representing the probability that the node will take a different value given different values ​​of its parent node. For workload state nodes, the probability distribution of different workload states can be estimated based on prior knowledge or historical data. For the corresponding blink frequency and average fixation time nodes in the visual channel data, the conditional probabilities of them taking different values ​​under different workload states need to be specified. These conditional probabilities can be obtained through expert knowledge or learned from the data. For example, the probability of frequent blinks and long average fixation time under high workload states can be estimated, as can the probability of normal blink frequency and short average fixation time under low workload states.

[0179] Through the above three steps, a Bayesian network model is constructed to represent the causal and conditional dependencies between each indirect evidence feature and the workload state.

[0180] Step S3: Perform Bayesian inference using the feature vector of indirect evidence.

[0181] Indirect evidence features are treated as observed variables, i.e., known conditions, and their corresponding child node values ​​are fixed in the Bayesian network. These child nodes include:

[0182] Visual channel data: [Gaze_duration_mean, Blink_frequency, Eye_closure]

[0183] Voice channel data: [Volume, F0, Speech_rate]

[0184] For each indirect evidence feature node, its posterior probability distribution in relation to the workload state is calculated using a Bayesian inference algorithm:

[0185] P(W|Gaze_duration_mean),P(W|Blink_frequency),P(W|Eye_closure);

[0186] P(W|Volume),P(W|F0),P(W|Speech_rate).

[0187] First, based on the structure of the Bayesian network and the conditional probability table (CPT), the posterior probability distribution of each indirect evidence feature node for the workload state needs to be calculated. Taking P(W|Gaze_duration_mean) as an example, the following needs to be calculated:

[0188] P(W|Gaze_duration_mean)=P(Gaze_duration_mean|W)*P(W) / P(Gaze_duration_mean)

[0189] in:

[0190] P(Gaze_duration_mean|W) is the likelihood probability of observing a specific average gaze duration given a workload state W. This probability can be obtained from the CPT of the Gaze_duration_mean node in the Bayesian network.

[0191] P(W) is the prior probability of the workload state W, which can be obtained from the preliminary evaluation result W_initial.

[0192] P(Gaze_duration_mean) is the marginal probability of the average gaze duration, which can be obtained by summing over the workload state W:

[0193] P(Gaze_duration_mean)=ΣP(Gaze_duration_mean|W)*P(W).

[0194] Where P(Gaze_duration_mean) = ΣP(Gaze_duration_mean|W)*P(W) is the marginal probability of the average gaze duration (Gaze_duration_mean) calculated according to the law of total probability. This formula means that the probability of observing a specific average gaze duration can be obtained by summing over all possible workload states W.

[0195] In one embodiment, the workload state W can take different values ​​(e.g., low load, medium load, high load).

[0196] For each possible workload state W, the conditional probability P(Gaze_duration_mean|W) of observing a specific average gaze duration in that state can be obtained from the conditional probability table (CPT) of the Bayesian network.

[0197] Meanwhile, the prior probability P(W) of each workload state W is obtained from the previous steps.

[0198] According to the law of total probability, the probability of observing a specific average fixation duration is equal to the sum of the products of the conditional probability and the prior probability under all possible workload states.

[0199] In a Bayesian network, each node has a conditional probability table (CPT) that describes the probabilistic relationship between the node and its parent node.

[0200] Taking the Gaze_duration_mean node as an example, assume the average gaze duration is divided into three intervals: short duration (<500ms), medium duration (500ms-1000ms), and long duration (>1000ms). Simultaneously, there are three workload states: low workload, medium workload, and high workload. Then, the CPT of the Gaze_duration_mean node is as follows:

[0201] Workload status: Short duration (<500ms), Medium duration (500ms-1000ms), Long duration (>1000ms)

[0202] Low load 0.7 0.2 0.1

[0203] Medium load 0.2 0.6 0.2

[0204] High load 0.1 0.2 0.7

[0205] This CPT means:

[0206] Under low load conditions, the probabilities of observing short, medium, and long durations are 0.7, 0.2, and 0.1, respectively.

[0207] Under medium load conditions, the probabilities of observing short, medium, and long durations were 0.2, 0.6, and 0.2, respectively.

[0208] Under high load conditions, the probabilities of observing short, medium, and long durations are 0.1, 0.2, and 0.7, respectively.

[0209] Similarly, corresponding CPTs can be constructed for other indirect evidence feature nodes (such as Blink_frequency, Eye_closure, Volume, F0, Speech_rate). These CPTs can be estimated based on expert knowledge, historical data, or experimental results.

[0210] Based on the CPT table above, P(Gaze_duration_mean) represents the probability distribution of the feature of average gaze duration, which consists of three parts: P(short duration), P(medium duration), and P(long duration). According to the law of total probability, we have:

[0211] P(Gaze_duration_mean) = P(short duration) + P(medium duration) + P(long duration)

[0212] Next, each part is computed using the given prior probabilities and CPT table:

[0213] Calculate P (short duration):

[0214] P(short duration) = P(short duration | low load) * P(low load) + P(short duration | medium load) * P(medium load) + P(short duration | high load) * P(high load)

[0215] = 0.7*0.5 + 0.2*0.3 + 0.1*0.2

[0216] =0.35 + 0.06 + 0.02

[0217] =0.43

[0218] Calculate P (medium duration):

[0219] P(Medium Duration) = P(Medium Duration | Low Load) * P(Low Load) + P(Medium Duration | Medium Load) * P(Medium Load) + P(Medium Duration | High Load) * P(High Load)

[0220] = 0.2*0.5 + 0.6*0.3 + 0.2*0.2

[0221] = 0.1 + 0.18 + 0.04

[0222] =0.32

[0223] Calculate P (duration):

[0224] P(Long Duration) = P(Long Duration | Low Load) * P(Low Load) + P(Long Duration | Medium Load) * P(Medium Load) + P(Long Duration | High Load) * P(High Load)

[0225] = 0.1*0.5 + 0.2*0.3 + 0.7*0.2

[0226] =0.05 + 0.06 + 0.14

[0227] =0.25

[0228] Therefore, the probability distribution of P(Gaze_duration_mean) is:

[0229] P(Gaze_duration_mean = shortest duration) = 0.43

[0230] P(Gaze_duration_mean = medium duration) = 0.32

[0231] P(Gaze_duration_mean = duration) = 0.25

[0232] That is, based on the given prior probabilities and the CPT table, we obtain:

[0233] The marginal probability of observing a short duration (<500ms) is 0.43 (43%).

[0234] The marginal probability of observing a medium duration (500ms-1000ms) is 0.32 (32%).

[0235] The marginal probability of observing a long duration (>1000ms) is 0.25 (25%).

[0236] The purpose of calculating marginal probabilities is to update the understanding of workload status based on observed evidence (such as average gaze duration) during Bayesian inference. By calculating P(Gaze_duration_mean), the probability of observing a specific average gaze duration can be obtained, and it can be used to calculate the posterior probability P(W|Gaze_duration_mean).

[0237] Similarly, the posterior probability distribution of other indirect evidence feature nodes for workload states can be calculated:

[0238] P(W|Blink_frequency), P(W|Eye_closure), P(W|Volume), P(W|F0), P(W|Speech_rate)

[0239] To calculate these posterior probability distributions, the CPT of the corresponding node needs to be obtained from the Bayesian network and calculated using the prior probability P(W) and the marginal probability.

[0240] Optionally, the workload status has three levels: low load, medium load, and high load.

[0241] For example, the posterior probability distribution obtained through Bayesian inference is shown below:

[0242] P(W|Gaze_duration_mean):

[0243] P(low load|Gaze_duration_mean)=0.6

[0244] P(Medium load|Gaze_duration_mean)=0.3

[0245] P(high load|Gaze_duration_mean)=0.1

[0246] P(W|Blink_frequency):

[0247] P(low load | Blink_frequency) = 0.7

[0248] P(medium load | Blink_frequency) = 0.2

[0249] P(high load|Blink_frequency)=0.1

[0250] P(W|Eye_closure):

[0251] P(Low load | Eye_closure) = 0.5

[0252] P(medium load | Eye_closure) = 0.3

[0253] P(High Load | Eye_closure) = 0.2

[0254] P(W|Volume):

[0255] P(Low Load | Volume) = 0.6

[0256] P(Medium Load|Volume)=0.3

[0257] P(High Load|Volume)=0.1

[0258] P(W|F0):

[0259] P(Low Load|F0) = 0.7

[0260] P(medium load|F0)=0.2

[0261] P(high load|F0)=0.1

[0262] P(W|Speech_rate):

[0263] P(low load|Speech_rate) = 0.6

[0264] P(Medium load|Speech_rate)=0.3

[0265] P(high load|Speech_rate)=0.1

[0266] Then, a weighted average was performed on the visual channel data and the audio channel data respectively to obtain the posterior probability distributions of the two channels:

[0267] P(W|Visual_Evidence)=(P(W|Gaze_duration_mean)+P(W|Blink_frequency)+P(W|Eye_closure)) / 3

[0268] P(W|Audio_Evidence)=(P(W|Volume)+P(W|F0)+P(W|Speech_rate)) / 3

[0269] For example, for the posterior probability distribution obtained through Bayesian inference, the features corresponding to the visual channel data and the speech channel data are weighted and averaged separately to obtain the posterior probability distributions of the two channels, specifically:

[0270] The posterior probability distribution within each channel is arithmetically averaged.

[0271] Weighted average of visual channels:

[0272] For low load conditions, we have:

[0273] P(Low Load | Visual Evidence) = (P(Low Load | Gaze_duration_mean) + P(Low Load | Blink_frequency) + P(Low Load | Eye_closure)) / 3

[0274] = (0.6 + 0.7 + 0.5) / 3

[0275] =1.8 / 3

[0276] =0.6

[0277] Similarly, for medium and high load conditions:

[0278] P(Medium Load | Visual Evidence) = (0.3 + 0.2 + 0.3) / 3 = 0.8 / 3 ≈ 0.27

[0279] P(High Load | Visual Evidence) = (0.1 + 0.1 + 0.2) / 3 = 0.4 / 3 ≈ 0.13

[0280] Weighted average of voice channels:

[0281] For low load conditions, we have:

[0282] P(Low Load | Audio_Evidence) = (P(Low Load | Volume) + P(Low Load | F0) + P(Low Load | Speech_rate)) / 3

[0283] = (0.6 + 0.7 + 0.6) / 3

[0284] =1.9 / 3

[0285] ≈0.63

[0286] Similarly, for medium and high load conditions:

[0287] P(Medium Load | Audio Evidence) = (0.3 + 0.2 + 0.3) / 3 = 0.8 / 3 ≈ 0.27

[0288] P(High Load | Audio_Evidence) = (0.1 + 0.1 + 0.1) / 3 = 0.3 / 3 = 0.1

[0289] This weighted average yielded the overall posterior probability distributions for the visual and audio channels:

[0290] P(W|Visual_Evidence):

[0291] P(Low Load | Visual Evidence) ≈ 0.6

[0292] P(Medium Load | Visual Evidence) ≈ 0.27

[0293] P(High Load | Visual Evidence) ≈ 0.13

[0294] P(W|Audio_Evidence):

[0295] P(Low Load | Audio_Evidence) ≈ 0.63

[0296] P(Medium load|Audio_Evidence)≈0.27

[0297] P(High Load|Audio_Evidence) = 0.1

[0298] Next, normalized quality weights W_visual_norm and W_audio_norm for the visual and audio channel data are introduced to weight and combine the posterior probability distributions of the two channels, resulting in the final posterior probability distribution P(W|Indirect_Evidence):

[0299] P(W|Indirect_Evidence)=W_visual_norm*P(W|Visual_Evidence)+W_audio_norm*P(W|Audio_Evidence),

[0300] Among them, W_visual_norm and W_audio_norm satisfy:

[0301] W_visual_norm+W_audio_norm=1

[0302] 0≤W_visual_norm,W_audio_norm≤1

[0303] Finally, based on the posterior probability distribution P(W|Indirect_Evidence) obtained by weighted combination, the workload state with the highest probability (low load, medium load, or high load) is selected as the final evaluation result W_final.

[0304] Step 4: Present the workload assessment results in a visual format to intuitively display the crew's real-time workload level. When the workload index exceeds a preset threshold, trigger an early warning to remind crew members and managers to pay attention to the risk of fatigue.

[0305] The dashboard displays the workload index for each crew member. The dashboard may include the following elements: color-coded workload levels (e.g., green for low workload, yellow for medium workload, and red for high workload), and basic crew information.

[0306] When the crew's workload indicator turns red, an early warning is triggered to remind crew members and managers to be aware of the risk of fatigue.

[0307] This invention proposes a multi-source data fusion-based method for assessing crew workload. By comprehensively utilizing heterogeneous data from multiple sources, it achieves a comprehensive and accurate assessment of crew workload status and real-time visualization, and establishes an effective early warning mechanism. This method has significant application value in improving crew work safety and operational efficiency. This invention effectively solves the technical bottlenecks of traditional workload assessment methods and demonstrates the innovative application of multi-source data fusion technology in the field of intelligent shipping.

[0308] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0309] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0310] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0311] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0312] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0313] The preferred embodiments of the present invention have been described above to make the spirit of the present invention clearer and easier to understand, and are not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope summarized by the appended claims.

Claims

1. A method for assessing crew workload based on multi-source data fusion, the method comprising: Step 1: Collect multi-source data, including physiological data and sensory monitoring data; Step 2: Extract features from the multi-source data; Step 3: Based on physiological data and sensory monitoring data, feature layer fusion and decision layer fusion are performed; Feature layer fusion and decision layer fusion include: Heart rate and electromyography data are used as direct evidence, while sensory monitoring data from the visual and auditory channels are used as indirect evidence. The features of heart rate and electromyography data are used to output the corresponding workload levels based on the trained workload classifier, and the preliminary workload assessment results are formed after feature fusion as the prior P(W) for subsequent inference. Construct a Bayesian network model with workload status as the parent node and visual channel features and voice channel features as child nodes; After using indirect evidence features as observation variables, the values ​​of the corresponding child nodes are fixed in the Bayesian network. Then, Bayesian inference is performed based on the prior P(W) to obtain the posterior probability distributions of the visual channel and the voice channel for the workload state. The posterior results of the two channels are weighted and combined to obtain the final posterior probability distribution, and the final workload assessment result is determined based on the maximum posterior probability. Step 4: Visualize the workload assessment results to show the crew's real-time workload level. When the workload index exceeds the preset threshold, trigger an early warning to remind crew members and managers to pay attention to the risk of fatigue.

2. The method for assessing crew workload based on multi-source data fusion as described in claim 1, characterized in that, The physiological data collection includes the use of portable physiological monitoring devices to collect real-time physiological data such as the crew's heart rate and electromyography. The sensory monitoring data includes video and audio sensors deployed in the work area to record crew members' facial expressions and voice data.

3. The method for assessing crew workload based on multi-source data fusion as described in claim 2, characterized in that, Fatigue monitoring data collection is achieved through cross-sensory coordination via visual and auditory channels; The visual channel includes capturing facial images and eye movement data of crew members through cameras for facial expression recognition and eye-tracking analysis; The auditory channel includes collecting voice data from crew members via microphones for voice emotion calculation and fatigue assessment.

4. The method for assessing crew workload based on multi-source data fusion as described in claim 1, characterized in that, The extraction of physiological data features from crew members includes at least: Analyze heart rate data to extract features such as mean heart rate and heart rate variability; Time-frequency analysis was performed on electromyography data to extract features related to muscle fatigue. For the visual channel, computer vision algorithms are used to extract facial expression features and eye movement features, including eye closure. Furthermore, eye-tracking pattern analysis was performed on the eye-tracking data to extract features such as fixation time and blink frequency. For the auditory channel, a speech emotion computing model is used to extract speech emotion features, which include volume, pitch, and speech rate.

5. The method for assessing crew workload based on multi-source data fusion as described in claim 1, characterized in that, Feature layer fusion employs adaptive feature weighting; in adaptive feature weighting, adaptive feature weighting coefficients are determined for physiological data and sensory monitoring data; by analyzing the signal-to-noise ratio, data quality, and correlation with workload of each data source, the weighting coefficients of each data source are dynamically adjusted.

6. The method for assessing crew workload based on multi-source data fusion as described in claim 1, characterized in that, Use machine learning models to train workload classifiers or regression models, with [HR_mean, SDNN, RMSSD] and [RMS, MPF] as input features, and output workload levels W_1 and W_2 respectively. in, HR_mean, SDNN, and RMSSD are used to extract the mean heart rate, SDNN, and RMSSD from the heart rate data, respectively. RMS and MPF are the root mean square value and average power frequency extracted from electromyography data, respectively. ， Where W_HR_norm and W_EMG_norm are the normalized quality weights of heart rate data and electromyography data obtained in step 2, respectively. W_1 and W_2 are the workload levels determined based on heart rate and electromyography data, respectively. The weighted result is denoted as the initial workload assessment result W_initial, and used as the prior probability P(W) in subsequent Bayesian inference.

7. The method for assessing crew workload based on multi-source data fusion as described in claim 1, characterized in that, Construct a Bayesian network model where nodes represent workload states and indirect evidence types, and edges represent dependencies between them. Assign a conditional probability table CPT to each node, representing the probability that the node takes a different value when its parent node takes a different value; Indirect evidence features are used as observation variables, i.e. known conditions, and the values ​​of the corresponding child nodes are fixed in the Bayesian network. For each indirect evidence feature node, the posterior probability distribution of its relation to the workload state is calculated using the Bayesian inference algorithm. We take a weighted average of the features corresponding to the visual channel data and the speech channel data respectively to obtain the posterior probability distribution of the two channels. The posterior probability distributions of the two channels are weighted and combined to obtain the final posterior probability distribution. Finally, based on the posterior probability distribution obtained by weighted combination, the workload state with the highest probability is selected as the final evaluation result. The workload states include: low load, medium load, or high load.

8. A crew workload assessment system based on multi-source data fusion, the system comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program in the memory to perform the method of any one of claims 1-7.

Images