Seaman workload assessment method and system based on multi-source data fusion

Abstract

The invention discloses a crew workload evaluation method and system based on multi-source data fusion, and the method comprises the steps: 1, carrying out the collection of multi-source data, and enabling the multi-source data to comprise objective data and sensory monitoring data; 2, performing feature extraction on the multi-source data; 3, based on the physiological data and the sensory monitoring data, performing feature layer fusion and decision-making layer fusion; and step 4, visualizing the workload evaluation result to present the real-time workload level of the sailors, and when the workload index exceeds a preset threshold value, triggering an early warning prompt to remind the sailors and managers to pay attention to the fatigue risk. According to the method, multi-source heterogeneous data are comprehensively utilized, comprehensive and accurate evaluation and real-time visual presentation of the load state of the sailors are achieved, an effective early warning mechanism is constructed, and the method has important application value in the aspect of improving the working safety and the working efficiency of the sailors.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a crew workload assessment method and system based on multi-source data fusion. Background Art

[0002] With the large-scale, intelligent and complex modern ships, the work intensity and psychological pressure of crew members are increasing day by day. Long-term high-intensity work can easily lead to fatigue and lack of concentration of crew members, which in turn leads to operational errors and misjudgment, which seriously threaten the safety of ship navigation and operation. Therefore, timely, accurate and comprehensive assessment of crew workload is of great significance for preventing accidents, protecting crew health and improving work efficiency.

[0003] Traditional crew workload assessment methods mainly rely on subjective self-assessment scales, such as the fatigue self-assessment scale and the NASA-TLX task load index. Although these methods are simple to operate, they are easily affected by the subjective consciousness of the crew, and the objectivity and reliability of the assessment results are difficult to guarantee. In recent years, with the development of emerging technologies such as the Internet of Things and artificial intelligence, using smart wearable devices to collect physiological data of crew members and analyzing the workload status of crew members through machine learning algorithms has become a new technical trend.

[0004] The present invention is proposed in this context, aiming to overcome the limitations of traditional methods and provide a crew workload assessment method based on multi-source data fusion, so as to more accurately, comprehensively and timely grasp the crew workload level, effectively prevent safety hazards caused by overwork, and ensure the health of crew members and the safety of ship operations. Summary of the invention

[0005] In view of the defects existing in the above-mentioned prior art, the present invention provides a crew workload assessment method based on multi-source data fusion, the method comprising:

[0006] Step 1: Collect multi-source data, where the multi-source data includes objective data and sensory monitoring data.

[0007] Step 2: extracting features from the multi-source data;

[0008] Step 3: Based on physiological data and sensory monitoring data, feature layer fusion and decision layer fusion are performed;

[0009] Step 4: Visualize the workload assessment results to present the crew's real-time workload level, and when the workload index exceeds the preset threshold, trigger an early warning prompt to remind the crew and managers to pay attention to fatigue risks.

[0010] The objective data collection includes using portable physiological monitoring equipment to collect the crew's heart rate and electromyography physiological data in real time.

[0011] The sensory monitoring data includes deploying video and audio sensors in the operating area to record the crew's facial expressions and voice data.

[0012] Among them, cross-sensory collaborative fatigue monitoring data collection is achieved through visual channels and auditory channels.

[0013] The visual channel includes collecting facial images and eye movement data of crew members through cameras for facial expression recognition and eye tracking analysis.

[0014] The auditory channel includes collecting crew voice data through microphones for voice emotion calculation and fatigue status judgment.

[0015] The extraction of physiological data features of crew members at least includes:

[0016] The heart rate data is analyzed to extract the characteristics of heart rate mean and heart rate variability.

[0017] Time-frequency analysis is performed on the electromyographic data to extract features related to muscle fatigue degree.

[0018] For the visual channel, a computer vision algorithm is used to extract facial expression features and eye movement features, wherein the facial expression features include eye closure degree;

[0019] And, analyze the eye movement patterns of the eye movement data to extract the features of fixation time and blink frequency.

[0020] For the auditory channel, a speech emotion calculation model is used to extract speech emotion features, which include volume, pitch, and speaking speed.

[0021] Among them, feature layer fusion performs adaptive feature weighting. In adaptive feature weighting, adaptive feature weighting coefficients are determined for different data sources such as physiological data and sensory monitoring data. By analyzing the signal-to-noise ratio, data quality and correlation with workload of each data source, the weight coefficient of each data source is dynamically adjusted.

[0022] Among them, the decision-level fusion is based on evidence theory, which regards physiological data as direct evidence and sensory monitoring data as indirect evidence. Through the combination of evidence, the probability distribution of the comprehensive workload state is obtained.

[0023] Among them, preliminary workload assessment is performed based on direct evidence feature vectors.

[0024] A Bayesian network model between workload influencing factors and workload status is constructed.

[0025] Bayesian inference using indirect evidence feature vectors.

[0026] Among them, a machine learning model is used to train a workload classifier or regression model, with [HR_mean, SDNN, RMSSD] and [RMS, MPF]] as input features, and the workload levels W_1 and W_2 are output respectively.

[0027] in,

[0028] HR_mean, SDNN, and RMSSD are the heart rate mean, SDNN, and RMSSD extracted from the heart rate data, respectively;

[0029] RMS and MPF are the root mean square value and mean power frequency extracted from EMG data, respectively;

[0030] W_initial=W_HR_norm*W_1+W_EMG_norm*W_2,

[0031] Wherein, W_HR_norm and W_EMG_norm are the normalized quality weights of heart rate data and electromyography data obtained in step 2, respectively;

[0032] W_1 and W_2 are the workload levels determined based on heart rate and electromyography data, respectively.

[0033] The weighted result is recorded as the initial workload assessment result W_initial, which serves as the prior probability P(W) in subsequent Bayesian inference.

[0034] A Bayesian network model is constructed, in which nodes represent workload status and indirect evidence types, and edges represent the dependencies between them.

[0035] A conditional probability table CPT is assigned to each node, which indicates the probability that the node takes different values ​​when its parent node takes different values.

[0036] The indirect evidence features are taken as observed variables, that is, known conditions, and the values ​​of the corresponding child nodes are fixed in the Bayesian network.

[0037] For each indirect evidence feature node, a Bayesian inference algorithm is used to calculate its posterior probability distribution for the workload state.

[0038] The features corresponding to the visual channel data and the speech channel data are weighted averaged to obtain the posterior probability distribution of the two channels;

[0039] Perform a weighted combination of the posterior probability distributions of the two channels to obtain the final posterior probability distribution;

[0040] Finally, according to the posterior probability distribution obtained by weighted combination, the workload state with the highest probability is selected as the final evaluation result;

[0041] The workload status includes: low load, medium load or high load.

[0042] The present invention also discloses a crew workload assessment system based on multi-source data fusion, the system comprising a memory and a processor, the memory storing a computer program, and the processor being used to execute the computer program in the memory to execute the aforementioned method.

[0043] The crew workload assessment method based on multi-source data fusion proposed in the present invention realizes comprehensive and accurate assessment and real-time visualization of crew load status by comprehensively utilizing multi-source heterogeneous data, and builds an effective early warning mechanism, which has important application value in improving crew work safety and operating efficiency. The invention solves the technical bottleneck of traditional load assessment methods and embodies the innovative application of multi-source data fusion technology in the field of intelligent shipping. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0045] Figure 1 The flowchart shows a crew workload assessment method based on multi-source data fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0048] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0049] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0050] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0051] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0052] The present invention proposes a crew workload assessment method based on multi-source data fusion, which comprehensively utilizes physiological data, cross-sensory collaborative immersive fatigue monitoring data and multi-perspective workload assessment data, and realizes comprehensive and accurate crew workload assessment through multi-level data fusion.

[0053] The present invention comprehensively utilizes multi-source data such as objective data and sensory monitoring data, overcomes the limitations of a single data source, evaluates the workload level of crew members from multiple dimensions such as physiology, behavior, and environment, and improves the accuracy and reliability of the evaluation results. The method combines feature layer fusion and decision layer fusion to process the complementarity and redundancy of multi-source data at the data level and decision level, respectively, give play to the advantages of different data sources, extract more comprehensive and refined load features, and obtain more robust evaluation results. The workload evaluation results are presented in real time in a visual manner, intuitively reflecting the fatigue state of the crew, facilitating the self-adjustment of the crew and real-time monitoring by managers, and effectively preventing safety accidents caused by fatigue driving. The warning threshold of the workload index is set, and when the load level of the crew exceeds the threshold, the warning prompt is automatically triggered, and the crew is reminded to take rest through sound and light, and the manager is notified to take necessary intervention measures, so as to effectively improve the safety of ship operations.

[0054] like Figure 1 As shown, the present invention proposes a crew workload assessment method based on multi-source data fusion, comprising:

[0055] Step 1: Collect multi-source data, including objective data and sensory monitoring data.

[0056] Among them, objective data collection includes the use of portable physiological monitoring equipment to collect the crew's heart rate and electromyography physiological data in real time.

[0057] The sensory monitoring data requires the deployment of video and audio sensors in the key or main operating areas of the ship to record the facial expressions and voice data of the crew. Specifically, cross-sensory collaborative fatigue monitoring data collection is achieved through visual channels and auditory channels.

[0058] Visual channel: The facial images and eye movement data of the crew members are collected through the camera for facial expression recognition and eye movement tracking analysis.

[0059] Auditory channel: The crew’s voice data is collected through a microphone for voice emotion calculation and fatigue status judgment.

[0060] The present invention evaluates crew workload based on multi-source data fusion, wherein data from physiological monitoring devices worn by crew members are collected to infer crew workload based on intelligent algorithms. In addition, the collected video and audio sensor data are used to prove the inference of the intensity of crew workload.

[0061] Step 2: extract features from the multi-source data.

[0062] Before feature extraction, the collected multi-source heterogeneous data are preprocessed with timestamp alignment, data cleaning, missing value interpolation and other preprocessing operations to improve data quality and consistency.

[0063] The extraction of physiological data features of crew members at least includes:

[0064] The heart rate data is analyzed to extract the characteristics of heart rate mean and heart rate variability.

[0065] Time-frequency analysis is performed on the electromyographic data to extract features related to muscle fatigue degree.

[0066] Among them, the heart rate data is analyzed to extract the characteristics of the heart rate mean and heart rate variability.

[0067] The mean heart rate indicates the average heart rate level over a period of time, reflecting the overall heart rate status of the crew.

[0068] HR_mean=Σ(HR_i) / N,

[0069] Among them, HR_i is the i-th heart rate data point, and N is the total number of heart rate data points.

[0070] Heart rate variability measures the degree of change of heart rate over a period of time and reflects the heart rate regulation ability of the crew. The heart rate variability indicators include SDNN (standard deviation of all RR intervals) and RMSSD (root mean square of differences between adjacent RR intervals).

[0071] SDNN=sqrt(Σ((RR_i-RR_mean)^2) / (N-1)),

[0072] RMSSD=sqrt(Σ((RR_i+1-RR_i)^2) / (N-1)),

[0073] Among them, RR_i is the i-th RR interval (the time interval between adjacent heartbeats), RR_mean is the average value of the RR interval, and N is the total number of RR intervals.

[0074] Among them, the time-frequency analysis of electromyographic data is performed to extract the characteristics related to the degree of muscle fatigue.

[0075] The root mean square value (RMS) reflects the energy level of the electromyographic signal and can be used to evaluate the intensity of muscle activity and fatigue.

[0076] RMS=sqrt(Σ(EMG_i^2) / N),

[0077] Among them, EMG_i is the i-th electromyographic data point, and N is the total number of electromyographic data points.

[0078] The mean power frequency MPF is the center frequency of the EMG power spectrum, and can also reflect the spectrum changes caused by muscle fatigue.

[0079] MPF=Σ(f_i*P_i) / Σ(P_i)

[0080] Among them, f_i is the center frequency of the i-th frequency bin, and P_i is the power spectral density value of the i-th frequency bin.

[0081] Among them, the feature extraction of crew sensory monitoring data includes:

[0082] For the visual channel, computer vision algorithms are used to extract facial expression features and eye movement features, such as eye closure, gaze direction, etc. Eye movement pattern analysis is performed on the eye movement data to extract features of gaze time and blink frequency.

[0083] For the auditory channel, the speech emotion computing model is used to extract speech emotion features such as volume, pitch, and speaking speed.

[0084] Among them, for the visual channel, computer vision algorithms are used to extract facial expression features and eye movement features.

[0085] Facial key points are detected using a facial key point detection algorithm, and then expression features are calculated based on the position and movement of the key points.

[0086] Based on the detected facial key points, the expression features are calculated.

[0087] The changes in facial expressions can be quantified by calculating geometric features such as the distance, angle, and area between key points. For example, by calculating the distance between the key points of the upper and lower eyelids, the degree of eye opening can be estimated; by calculating the position change of the corners of the mouth, the expression of smile or surprise can be determined.

[0088] Among them, the eye movement data is analyzed for eye movement patterns to extract the features of gaze time and blinking frequency.

[0089] Eye movement pattern analysis uses the eye key point data in the facial key points to perform eye movement pattern analysis, including: extracting the key point coordinates of the left and right eyes, calculating the center position of the eyes, tracking the changes in the center position of the eyes, obtaining the eye movement trajectory, and calculating the fixation time and blinking frequency based on the eye movement trajectory.

[0090] Among them, the average gaze time Gaze_duration_mean represents the average gaze duration of the crew in the corresponding area of ​​interest.

[0091] Gaze_duration_mean=Σ(Gaze_duration_i) / N

[0092] Where Gaze_duration_i is the duration of the i-th gaze, and N is the total number of gazes.

[0093] Blink_frequency refers to the number of blinks per unit time, which reflects the crew's fatigue level and attention level.

[0094] Blink_frequency=N_blink / T,

[0095] Among them, N_blink is the total number of blinks, and T is the total measurement time.

[0096] Expression features include eye closure, which is determined by calculating the ratio of the distance between the upper and lower eyelids to the width of the eye.

[0097] Eye_closure=(P_upper-P_lower) / Eye_width,

[0098] Among them, P_upper and P_lower are the vertical coordinates of the key points of the upper and lower eyelids, and Eye_width is the width of the eye.

[0099] Among them, for the auditory channel, the speech emotion calculation model is used to extract speech emotion features.

[0100] First, the recorded voice data is preprocessed, such as removing silent segments and reducing noise, to improve the voice quality.

[0101] Then, a speech emotion recognition model is used, such as a speech feature extraction model based on SVM, DNN and other algorithms, to extract the emotion features of the speech. The emotion features specifically include:

[0102] 1. Volume:

[0103] Calculate the root mean square (RMS) value of the speech signal to indicate the loudness of the speech.

[0104] Volume = sqrt(Σ(x_i^2) / N),

[0105] Among them, x_i is the i-th sampling point of the speech signal, and N is the total number of sampling points.

[0106] 2. Tone:

[0107] Calculate the fundamental frequency (F0) of the speech signal, which represents the pitch of the speech.

[0108] The fundamental frequency can be estimated using a time domain based algorithm (such as the YIN algorithm) or a frequency domain based algorithm (such as the Cepstrum algorithm).

[0109] 3. Speaking speed:

[0110] Calculate the number of syllables or words of speech per unit time to indicate the speed of speech.

[0111] Speech_rate=N_syllables / T,

[0112] Among them, N_syllables is the total number of syllables and T is the total duration of speech.

[0113] Step 3: Based on physiological data and sensory monitoring data, perform feature layer fusion and decision layer fusion.

[0114] Among them, feature layer fusion performs adaptive feature weighting.

[0115] Among them, the decision-level fusion is based on evidence theory, which regards physiological data as direct evidence and sensory monitoring data as indirect evidence. Through the combination of evidence, the probability distribution of the comprehensive workload state is obtained.

[0116] Among them, in the adaptive feature weighting, the adaptive feature weighting coefficient is determined for different data sources such as physiological data and sensory monitoring data. By analyzing the signal-to-noise ratio, data quality and correlation with workload of each data source, the weight coefficient of each data source is dynamically adjusted.

[0117] Among them, physiological data quality assessment is the process of quality analysis of the collected heart rate, electromyography and eye movement data. The quality of each type of physiological data source can be quantified by calculating indicators such as signal integrity, signal-to-noise ratio, data validity and calibration accuracy.

[0118] For heart rate data and electromyography data, signal integrity measures the data missing during the data acquisition process. By calculating the ratio of missing data points to the total number of data points, the integrity of the signal can be evaluated.

[0119] Among them, missing rate = number of missing data points / total number of data points. Missing data points means the number of data points that were not successfully collected, and total data points means the total number of data points collected.

[0120] And, for heart rate data and electromyography data, the signal-to-noise ratio measures the ratio of useful information to noise in the signal. By calculating the ratio of signal power to noise power, the quality of the signal can be evaluated.

[0121] SNR = 10*log10 (signal power / noise power),

[0122] Signal power = Σ(x_i^2) / N, where x_i is the signal amplitude and N is the signal length;

[0123] Noise power = Σ(n_i^2) / N, where n_i is the noise amplitude and N is the noise length;

[0124] Signal power represents the average energy of the signal, and noise power represents the average energy of the noise.

[0125] Among them, sensory monitoring data quality assessment is the process of quality analysis of the collected facial expressions, eye movements (visual channel) and voice data.

[0126] Among them, image clarity measures the clarity of facial expression images. The clarity of an image can be evaluated by calculating the sum of the absolute values ​​of the gradients between adjacent pixels in the image.

[0127] Clarity=Σ(abs(I(x,y)-I(x+1,y))+abs(I(x,y)-I(x,y+1))) / (2*W*H),

[0128] Where I(x,y) is the pixel value of the image at position (x,y), and W and H are the width and height of the image respectively.

[0129] The face detection rate measures the proportion of successfully detected faces in facial expression data. The effectiveness of face detection can be evaluated by calculating the proportion of frames with successfully detected faces to the total number of frames.

[0130] Detection rate = number of frames that successfully detect faces / total number of frames,

[0131] Among them, the number of frames in which faces are successfully detected indicates the number of frames in which the algorithm successfully recognizes faces, and the total number of frames indicates the total number of frames of facial expression data.

[0132] In addition, for eye movement data (visual channel), the pupil detection rate measures the proportion of pupils successfully detected in the eye movement data. The effect of pupil detection can be evaluated by calculating the proportion of frames in which pupils are successfully detected to the total number of frames.

[0133] Detection rate = number of frames in which pupils are successfully detected / total number of frames

[0134] Among them, the number of frames in which the pupil is successfully detected represents the number of frames in which the algorithm successfully identifies the pupil, and the total number of frames represents the total number of frames of eye movement data.

[0135] For features extracted from speech data, the signal-to-noise ratio measures the ratio of speech to noise in a speech signal. By calculating the ratio of speech signal power to noise power, the quality of the speech signal can be evaluated.

[0136] SNR = 10*log10 (speech signal power / noise power);

[0137] Speech signal power = Σ(s_i^2) / N, where s_i is the amplitude of the speech signal and N is the length of the speech signal;

[0138] Noise power = Σ(n_i^2) / N, where n_i is the noise amplitude and N is the noise length;

[0139] The speech signal power represents the average energy of the speech signal, and the noise power represents the average energy of the noise.

[0140] After that, in order to comprehensively evaluate the quality of each data source, it is necessary to convert each quality indicator into a unified quality score. By normalizing the quality indicators, indicators of different dimensions can be mapped to the same scale. Minimum-maximum normalization scales the value range of the quality indicator to the interval [0,1].

[0141] In order to dynamically adjust the weight of each data source, it is necessary to combine the data quality score and the initial weight for calculation. Through quality weight update and weight normalization, the final weight of each data source can be obtained.

[0142] Quality weight update multiplies the data quality score by the initial weight to obtain the weight that takes quality factors into consideration.

[0143] W_i=Q_i*W_i_init;

[0144] Among them, Wi is the quality weight of the i-th data source, Qi is the quality score of the i-th data source, and Wi_i_init is the initial weight of the i-th data source.

[0145] The quality scores of the data sources are obtained as follows:

[0146] Heart rate data quality score Q_HR:

[0147] Q_HR=(1-missing rate_HR)*α_HR+SNR_HR*β_HR,

[0148] Among them, missing rate_HR and SNR_HR are the missing rate and signal-to-noise ratio of heart rate data respectively, α_HR and β_HR are the weight coefficients of missing rate and signal-to-noise ratio, satisfying α_HR+β_HR=1.

[0149] EMG data quality score Q_EMG:

[0150] Q_EMG=(1-missing rate_EMG)*α_EMG+SNR_EMG*β_EMG,

[0151] Among them, missing rate_EMG and SNR_EMG are the missing rate and signal-to-noise ratio of electromyographic data respectively, α_EMG and β_EMG are the weight coefficients of missing rate and signal-to-noise ratio, satisfying α_EMG+β_EMG=1.

[0152] Visual channel data quality score Q_Visual:

[0153] Q_Visual=clarity_Visual*λ_Visual+face detection rate_Visual*μ_Visual+pupil detection rate_Visual*ν_Visual

[0154] Among them, clarity_Visual, face detection rate_Visual and pupil detection rate_Visual are the image clarity, face detection rate and pupil detection rate of the visual channel data respectively, λ_Visual, μ_Visual and ν_Visual are the weight coefficients of the three indicators, satisfying λ_Visual+μ_Visual+ν_Visual=1.

[0155] Auditory channel data quality score Q_Audio:

[0156] Q_Audio=SNR_Audio

[0157] Among them, SNR_Audio is the signal-to-noise ratio of voice data.

[0158] Furthermore, the weights corresponding to the direct evidence features and the indirect evidence features are normalized respectively by dividing the quality weight by the sum of all weights to ensure that the sum of the weights of the direct evidence features and the sum of the weights of the indirect evidence features are both 1.

[0159] W_HR_norm and W_EMG_norm are the normalized quality weights for the heart rate data and electromyography data in the direct evidence feature group, respectively.

[0160] W_visual_norm and W_Audio_norm are the normalized quality weights of the visual channel data and the audio channel data in the indirect evidence feature group respectively.

[0161] The decision-level fusion includes reasoning about the workload based on evidence theory.

[0162] Among them, physiological data is regarded as direct evidence, providing direct information about the workload status, while sensory monitoring data is regarded as indirect evidence, providing indirect information about the factors affecting the workload.

[0163] The direct evidence feature vector and the indirect evidence feature vector are used as evidence in the evidence theory framework, which respectively represent the support of physiological data and sensory monitoring data for the workload state. Specifically, they include:

[0164] Step S1: Perform preliminary workload assessment based on direct evidence feature vectors.

[0165] Use a machine learning model (such as support vector machine SVM) to train a workload classifier or regression model, taking [HR_mean, SDNN, RMSSD] and [RMS, MPF]] as input features and outputting workload levels W_1 and W_2 respectively.

[0166] in,

[0167] HR_mean, SDNN, and RMSSD are the features extracted from heart rate data: heart rate mean, SDNN, and RMSSD;

[0168] RMS and MPF are the features extracted from EMG data: root mean square value and mean power frequency.

[0169] W_initial=W_HR_norm*W_1+W_EMG_norm*W_2,

[0170] Wherein, W_HR_norm and W_EMG_norm are the normalized quality weights of heart rate data and electromyography data obtained in step 2, respectively;

[0171] W_1 and W_2 are the workload levels determined based on heart rate and electromyography data, respectively.

[0172] The weighted result is recorded as the initial workload assessment result W_initial, which serves as the prior probability P(W) in subsequent Bayesian inference.

[0173] Step S2: Construct a Bayesian network model between workload influencing factors and workload status.

[0174] Based on domain knowledge and data analysis, the causal relationship and conditional dependency between indirect evidence features and workload status are determined. A Bayesian network model is constructed, in which nodes represent workload status and indirect evidence types, and edges represent the dependency between them. For example, in the Bayesian network model, the workload status node is the parent node, and each indirect evidence feature is a child node.

[0175] Assign a conditional probability table (CPT) to each node, which indicates the probability of the node taking different values ​​when its parent node takes different values. The CPT can be obtained through expert knowledge or learned from data.

[0176] Based on domain knowledge and data analysis, determine the causal relationship and conditional dependency between indirect evidence features and workload status. For example, it can be assumed that both blink frequency and average fixation time are affected by workload status, and that blink frequency and average fixation time are conditionally independent given the workload status. These assumptions can be verified based on expert knowledge or through data analysis.

[0177] A Bayesian network model is constructed, in which nodes represent workload status and indirect evidence types, and edges represent the dependencies between them. For example, in the Bayesian network model, the workload status node is the parent node, and the blink frequency node and the average fixation time node are child nodes.

[0178] A conditional probability table (CPT) is assigned to each node, indicating the probability of the node taking different values ​​when its parent node takes different values. For workload status nodes, the probability distribution of different workload states can be estimated based on prior knowledge or historical data. For the corresponding blink frequency nodes and average fixation time nodes in the visual channel data, the conditional probabilities of them taking different values ​​under different workload states need to be specified. These conditional probabilities can be obtained through expert knowledge or learned from data. For example, the probability of frequent blinking frequency and long average fixation time under high workload state can be estimated, and the probability of normal blinking frequency and short average fixation time under low workload state can be estimated.

[0179] Through the above three steps, a Bayesian network model is constructed to represent the causal relationship and conditional dependency between each indirect evidence feature and workload status.

[0180] Step S3: Perform Bayesian reasoning using the indirect evidence feature vector.

[0181] The indirect evidence features are used as observed variables, i.e. known conditions, and the values ​​of the corresponding child nodes are fixed in the Bayesian network. These child nodes include:

[0182] Visual channel data: [Gaze_duration_mean, Blink_frequency, Eye_closure]

[0183] Voice channel data: [Volume, F0, Speech_rate]

[0184] For each indirect evidence feature node, the Bayesian inference algorithm is used to calculate its posterior probability distribution for the workload state:

[0185] P(W|Gaze_duration_mean),P(W|Blink_frequency),P(W|Eye_closure);

[0186] P(W|Volume),P(W|F0),P(W|Speech_rate).

[0187] First, we need to calculate the posterior probability distribution of each indirect evidence feature node for the workload state based on the structure of the Bayesian network and the conditional probability table (CPT). Taking P(W|Gaze_duration_mean) as an example, we need to calculate:

[0188] P(W|Gaze_duration_mean)=P(Gaze_duration_mean|W)*P(W) / P(Gaze_duration_mean)

[0189] in:

[0190] P(Gaze_duration_mean|W) is the likelihood of observing a specific mean gaze duration given a workload state W. This probability can be obtained from the CPT of the Gaze_duration_mean node in the Bayesian network.

[0191] P(W) is the prior probability of workload state W, which can be obtained based on the initial evaluation result W_initial.

[0192] P(Gaze_duration_mean) is the marginal probability of the average gaze duration, which can be obtained by summing the workload states W:

[0193] P(Gaze_duration_mean)=ΣP(Gaze_duration_mean|W)*P(W).

[0194] Among them, P(Gaze_duration_mean) = ΣP(Gaze_duration_mean|W)*P(W) is the marginal probability of the average gaze duration (Gaze_duration_mean) calculated according to the total probability formula. This formula shows that the probability of observing a specific average gaze duration can be obtained by summing all possible workload states W.

[0195] In one embodiment, the workload state W may take different values ​​(eg, low load, medium load, high load).

[0196] For each possible workload state W, the conditional probability P(Gaze_duration_mean|W) of observing a specific mean gaze duration in this state can be obtained from the conditional probability table (CPT) of the Bayesian network.

[0197] At the same time, the prior probability P(W) of each workload state W is W_initial obtained in the previous step.

[0198] According to the total probability formula, the probability of observing a specific average fixation duration is equal to the sum of the product of the conditional probability and the prior probability under all possible workload states.

[0199] In a Bayesian network, each node has a conditional probability table (CPT) that describes the probability relationship between the node and its parent node.

[0200] Taking the Gaze_duration_mean node as an example, assume that the average gaze duration is divided into three intervals: short duration (<500ms), medium duration (500ms-1000ms) and long duration (>1000ms). At the same time, there are three workload states: low load, medium load and high load. Then, the CPT of the Gaze_duration_mean node is as follows:

[0201] Workload status Short duration (<500ms) Medium duration (500ms-1000ms) Long duration (>1000ms)

[0202] Low load 0.7 0.2 0.1

[0203] Medium load 0.2 0.6 0.2

[0204] High load 0.1 0.2 0.7

[0205] This CPT means:

[0206] Under low load conditions, the probabilities of observing short, medium, and long durations are 0.7, 0.2, and 0.1, respectively.

[0207] Under medium load conditions, the probabilities of observing short duration, medium duration, and long duration are 0.2, 0.6, and 0.2, respectively.

[0208] Under high load conditions, the probabilities of observing short, medium, and long durations are 0.1, 0.2, and 0.7, respectively.

[0209] Similarly, corresponding CPTs can be constructed for other indirect evidence feature nodes (such as Blink_frequency, Eye_closure, Volume, F0, Speech_rate). These CPTs can be estimated based on expert knowledge, historical data or experimental results.

[0210] Based on the above CPT table, P(Gaze_duration_mean) represents the probability distribution of the feature of average gaze duration, which consists of three parts: P(short duration), P(medium duration) and P(long duration). According to the total probability formula, we have:

[0211] P(Gaze_duration_mean) = P(short duration) + P(medium duration) + P(long duration)

[0212] Next, use the given prior probabilities and the CPT table to calculate each part:

[0213] Calculate P(short duration):

[0214] P(short duration) = P(short duration | low load) * P(low load) + P(short duration | medium load) * P(medium load) + P(short duration | high load) * P(high load)

[0215] =0.7*0.5+0.2*0.3+0.1*0.2

[0216] =0.35+0.06+0.02

[0217] =0.43

[0218] Calculate P(medium duration):

[0219] P(medium duration) = P(medium duration | low load) * P(low load) + P(medium duration | medium load) * P(medium load) + P(medium duration | high load) * P(high load)

[0220] =0.2*0.5+0.6*0.3+0.2*0.2

[0221] =0.1+0.18+0.04

[0222] =0.32

[0223] Calculate P(long duration):

[0224] P(long duration) = P(long duration | low load) * P(low load) + P(long duration | medium load) * P(medium load) + P(long duration | high load) * P(high load)

[0225] =0.1*0.5+0.2*0.3+0.7*0.2

[0226] =0.05+0.06+0.14

[0227] =0.25

[0228] Therefore, the probability distribution of P(Gaze_duration_mean) is:

[0229] P(Gaze_duration_mean=short duration)=0.43

[0230] P(Gaze_duration_mean=Medium duration)=0.32

[0231] P(Gaze_duration_mean=long duration)=0.25

[0232] That is, according to the given prior probability and CPT table, we get:

[0233] The marginal probability of observing short duration (<500ms) is 0.43 (43%)

[0234] The marginal probability of observing a medium duration (500ms-1000ms) is 0.32 (32%)

[0235] The marginal probability of observing a long duration (>1000ms) is 0.25 (25%)

[0236] The purpose of calculating marginal probabilities is to update the understanding of workload status based on observed evidence (such as average gaze duration) during Bayesian inference. By calculating P(Gaze_duration_mean), the probability of observing a specific average gaze duration can be obtained and used to calculate the posterior probability P(W|Gaze_duration_mean).

[0237] Similarly, the posterior probability distribution of other indirect evidence feature nodes to the workload status can be calculated:

[0238] P(W|Blink_frequency), P(W|Eye_closure), P(W|Volume), P(W|F0), P(W|Speech_rate)

[0239] When calculating these posterior probability distributions, it is necessary to obtain the CPT of the corresponding nodes from the Bayesian network and use the prior probability P(W) and marginal probability for calculation.

[0240] Optionally, there are three levels of workload status: low load, medium load and high load,

[0241] For example, the posterior probability distribution calculated by Bayesian inference is as follows:

[0242] P(W|Gaze_duration_mean):

[0243] P(low load|Gaze_duration_mean)=0.6

[0244] P(Medium Load|Gaze_duration_mean)=0.3

[0245] P(high load|Gaze_duration_mean)=0.1

[0246] P(W|Blink_frequency):

[0247] P(low load|Blink_frequency)=0.7

[0248] P(medium load|Blink_frequency)=0.2

[0249] P(high load|Blink_frequency)=0.1

[0250] P(W|Eye_closure):

[0251] P(low load|Eye_closure)=0.5

[0252] P(Medium Load|Eye_closure)=0.3

[0253] P(high load|Eye_closure)=0.2

[0254] P(W|Volume):

[0255] P(low load|Volume)=0.6

[0256] P(Medium Load|Volume)=0.3

[0257] P(high load|Volume)=0.1

[0258] P(W|F0):

[0259] P(low load|F0)=0.7

[0260] P(medium load|F0)=0.2

[0261] P(high load|F0)=0.1

[0262] P(W|Speech_rate):

[0263] P(low load|Speech_rate)=0.6

[0264] P(Medium Load|Speech_rate)=0.3

[0265] P(high load|speech_rate)=0.1

[0266] Afterwards, the visual channel data and the speech channel data are weighted averaged to obtain the posterior probability distribution of the two channels:

[0267] P(W|Visual_Evidence)=(P(W|Gaze_duration_mean)+P(W|Blink_frequency)+P(W|Eye_closure)) / 3

[0268] P(W|Audio_Evidence)=(P(W|Volume)+P(W|F0)+P(W|Speech_rate)) / 3

[0269] For example, for the posterior probability distribution result obtained by Bayesian reasoning, the features corresponding to the visual channel data and the speech channel data are weighted averaged to obtain the posterior probability distribution of the two channels, specifically:

[0270] The posterior probability distribution within each channel is arithmetic averaged.

[0271] Weighted average of visual channels:

[0272] For low load conditions, there are:

[0273] P(low load|Visual_Evidence)=(P(low load|Gaze_duration_mean)+P(low load|Blink_frequency)+P(low load|Eye_closure)) / 3

[0274] =(0.6+0.7+0.5) / 3

[0275] =1.8 / 3

[0276] =0.6

[0277] Similarly, for medium and high load conditions:

[0278] P(Medium Load|Visual_Evidence)=(0.3+0.2+0.3) / 3=0.8 / 3≈0.27

[0279] P(high load|Visual_Evidence)=(0.1+0.1+0.2) / 3=0.4 / 3≈0.13

[0280] Weighted average of speech channels:

[0281] For low load conditions, there are:

[0282] P(low load|Audio_Evidence)=(P(low load|Volume)+P(low load|F0)+P(low load|Speech_rate)) / 3

[0283] =(0.6+0.7+0.6) / 3

[0284] =1.9 / 3

[0285] ≈0.63

[0286] Similarly, for medium and high load conditions:

[0287] P(Medium Load|Audio_Evidence)=(0.3+0.2+0.3) / 3=0.8 / 3≈0.27

[0288] P(high load|Audio_Evidence)=(0.1+0.1+0.1) / 3=0.3 / 3=0.1

[0289] Through this weighted average, the overall posterior probability distribution of the visual channel and the speech channel is obtained:

[0290] P(W|Visual_Evidence):

[0291] P(low load|Visual_Evidence)≈0.6

[0292] P(Medium Load|Visual_Evidence)≈0.27

[0293] P(High Load|Visual_Evidence)≈0.13

[0294] P(W|Audio_Evidence):

[0295] P(low load|Audio_Evidence)≈0.63

[0296] P(Medium Load|Audio_Evidence)≈0.27

[0297] P(High Load|Audio_Evidence)=0.1

[0298] After that, the normalized quality weights W_visual_norm and W_audio_norm of the visual channel data and the speech channel data are introduced, and the posterior probability distributions of the two channels are weighted combined to obtain the final posterior probability distribution P(W|Indirect_Evidence):

[0299] P(W|Indirect_Evidence)=W_visual_norm*P(W|Visual_Evidence)+W_audio_norm*P(W|Audio_Evidence),

[0300] Among them, W_visual_norm and W_audio_norm satisfy:

[0301] W_visual_norm+W_audio_norm=1

[0302] 0≤W_visual_norm,W_audio_norm≤1

[0303] Finally, according to the posterior probability distribution P(W|Indirect_Evidence) obtained by weighted combination, the workload state with the highest probability (low load, medium load or high load) is selected as the final evaluation result W_final.

[0304] Step 4: Present the workload assessment results in a visual form to intuitively display the crew's real-time workload level. When the workload index exceeds the preset threshold, an early warning prompt is triggered to remind the crew and managers to pay attention to fatigue risks.

[0305] In the dashboard, the workload index of each crew member is displayed. The dashboard may include the following elements: color coding of workload level (such as green for low load, yellow for medium load, and red for high load), and basic information of the crew member.

[0306] When the crew's workload indicator turns red, an early warning prompt is triggered to remind the crew and managers to pay attention to fatigue risks.

[0307] The crew workload assessment method based on multi-source data fusion proposed in the present invention realizes comprehensive and accurate assessment and real-time visualization of crew load status by comprehensively utilizing multi-source heterogeneous data, and builds an effective early warning mechanism, which has important application value in improving crew work safety and operating efficiency. The invention solves the technical bottleneck of traditional load assessment methods and embodies the innovative application of multi-source data fusion technology in the field of intelligent shipping.

[0308] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0309] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0310] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0311] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0312] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0313] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A crew workload assessment method based on multi-source data fusion, the method comprising: Step 1: Collect multi-source data, where the multi-source data includes objective data and sensory monitoring data; Step 2: extracting features from the multi-source data; Step 3: Based on physiological data and sensory monitoring data, feature layer fusion and decision layer fusion are performed; Step 4: Visualize the workload assessment results to present the crew's real-time workload level, and when the workload index exceeds the preset threshold, trigger an early warning prompt to remind the crew and managers to pay attention to fatigue risks.

2. The crew workload assessment method based on multi-source data fusion according to claim 1, characterized in that: The objective data collection includes using portable physiological monitoring equipment to collect the crew's heart rate and electromyography physiological data in real time; The sensory monitoring data includes deploying video and audio sensors in the operating area to record the crew's facial expressions and voice data.

3. The crew workload assessment method based on multi-source data fusion according to claim 2, characterized in that: Cross-sensory collaborative fatigue monitoring data collection is achieved through visual and auditory channels; The visual channel includes collecting facial images and eye movement data of crew members through cameras for facial expression recognition and eye tracking analysis; The auditory channel includes collecting crew voice data through microphones for voice emotion calculation and fatigue status judgment.

4. The crew workload assessment method based on multi-source data fusion according to claim 1, characterized in that: Extraction of physiological data characteristics of crew members, including at least: Analyze the heart rate data and extract the characteristics of heart rate mean and heart rate variability; Conduct time-frequency analysis on electromyographic data to extract features related to muscle fatigue degree; For the visual channel, a computer vision algorithm is used to extract facial expression features and eye movement features, wherein the facial expression features include eye closure degree; In addition, eye movement pattern analysis is performed on the eye movement data to extract features of gaze time and blinking frequency. For the auditory channel, a speech emotion calculation model is used to extract speech emotion features, which include volume, pitch, and speaking speed.

5. The crew workload assessment method based on multi-source data fusion according to claim 1, characterized in that: Feature layer fusion is used to perform adaptive feature weighting. In adaptive feature weighting, adaptive feature weighting coefficients are determined for different data sources such as physiological data and sensory monitoring data. The weight coefficient of each data source is dynamically adjusted by analyzing the signal-to-noise ratio, data quality and correlation with workload of each data source.

6. The crew workload assessment method based on multi-source data fusion according to claim 1, characterized in that: The decision-level fusion is based on evidence theory, which regards physiological data as direct evidence and sensory monitoring data as indirect evidence. By combining the evidence, the probability distribution of the comprehensive workload state is obtained.

7. The crew workload assessment method based on multi-source data fusion according to claim 6, characterized in that: Conduct preliminary workload assessment based on direct evidence feature vectors; Construct a Bayesian network model between workload influencing factors and workload status; Bayesian inference using indirect evidence feature vectors.

8. The crew workload assessment method based on multi-source data fusion according to claim 7, characterized in that: Use machine learning models to train workload classifiers or regression models, taking [HR_mean, SDNN, RMSSD] and [RMS, MPF]] as input features, and output workload levels W_1 and W_2, respectively; in, HR_mean, SDNN, and RMSSD are the heart rate mean, SDNN, and RMSSD extracted from the heart rate data, respectively; RMS and MPF are the root mean square value and mean power frequency extracted from EMG data, respectively; W_initial=W_HR_norm*W_1+W_EMG_norm*W_2, Wherein, W_HR_norm and W_EMG_norm are the normalized quality weights of heart rate data and electromyography data obtained in step 2, respectively; W_1 and W_2 are workload levels determined based on heart rate and electromyography data, respectively; The weighted result is recorded as the initial workload assessment result W_initial, which serves as the prior probability P(W) in subsequent Bayesian inference.

9. The crew workload assessment method based on multi-source data fusion according to claim 7, characterized in that: A Bayesian network model is constructed, where nodes represent workload status and indirect evidence types, and edges represent the dependencies between them; Assign a conditional probability table CPT to each node, indicating the probability of the node taking different values ​​when its parent node takes different values; The indirect evidence features are taken as observed variables, i.e. known conditions, and the values ​​of the corresponding child nodes are fixed in the Bayesian network; For each indirect evidence feature node, the Bayesian inference algorithm is used to calculate its posterior probability distribution for the workload state; The features corresponding to the visual channel data and the speech channel data are weighted averaged to obtain the posterior probability distribution of the two channels; Perform a weighted combination of the posterior probability distributions of the two channels to obtain the final posterior probability distribution; Finally, according to the posterior probability distribution obtained by weighted combination, the workload state with the highest probability is selected as the final evaluation result; The workload status includes: low load, medium load or high load.

10. A crew workload assessment system based on multi-source data fusion, the system comprising a memory and a processor, the memory storing a computer program, the processor being configured to execute the computer program in the memory to execute the method according to any one of claims 1 to 9.

Images