Pilot fatigue state monitoring method based on credible conflict multimodal learning
Through the evidence neural network and Dilicre distribution parameters to deal with intermodal conflicts, the trusted conflict multimodal learning of pilot fatigue status monitoring is realized, the problem of neglecting intermodal conflicts in the existing technology is solved, and the detection results are improved. Trustworthiness and accuracy.
Patent Information
- Application Number
- CN202410189362.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-20
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-02-20
AI Technical Summary
The existing multimodal learning methods ignore the conflict information between modes in pilot fatigue state monitoring, resulting in unreliable detection of detection results, which may cause incorrect fatigue state and degree detection, thereby causing unnecessary trouble or even accidents.
Using a trusted conflict multimodal learning method based on evidence neural network, the pilot's physiological data is obtained by obtaining the pilot's multiple modalities, extracting features and assigning them to the fatigue degree category, generating views of each modality, and fusing views to calculate the pilot's fatigue degree Class probability and confidence, using Dirichlet distribution parameters and intensity to deal with intermodal conflicts.
This method can effectively aggregate conflict information between modes, improve the credibility and accuracy of detection results, avoid error detection, and enhance the reliability of pilot fatigue status monitoring.
Smart Images

Figure CN118228190B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of fatigue detection, and in particular relates to a pilot fatigue state monitoring method based on credible conflict multimodal learning. Background Art
[0002] At present, pilot status monitoring usually uses the collected data of multiple physiological modalities of pilots to analyze their fatigue level, such as: collecting EEG signals through wearable EEG caps, collecting RGB images through optical cameras, collecting infrared images through infrared cameras, collecting ECG signals through heart rate bracelets, etc. Among them, EEG signals, RGB images, infrared images, ECG signals and other data are called multimodal data. After obtaining the above multimodal data, the consistency and complementary information between different modalities are used to make a judgment on the pilot's driving fatigue level.
[0003] However, there may be certain conflicts between the information carried by different modalities. For example, the RGB image modality may give a non-fatigue decision due to insufficient light, and the heart rate modality may give a high degree of fatigue decision due to the pilot dealing with emergency matters. Although the traditional multimodal learning method simply integrates the decision opinions of each modality, it ignores the interference of conflicting information between modalities on the decision results. The conflicting information in different modalities will inevitably cause the multimodal learning method to degenerate or even fail, resulting in erroneous and unreliable detection results. The erroneous fatigue state and fatigue degree detection results will cause the anti-fatigue system to intervene to the wrong degree, which will cause unnecessary trouble to pilots and ground tower personnel, and even cause accidents. Summary of the invention
[0004] In order to solve the above problems existing in the prior art, the present invention provides a pilot fatigue state monitoring method based on credible conflict multimodal learning. The technical problem to be solved by the present invention is achieved through the following technical solutions:
[0005] The present invention provides a pilot fatigue state monitoring method based on credible conflict multimodal learning, comprising:
[0006] Acquire the pilot's physiological data in multiple modes and perform preprocessing to obtain the characteristics of each mode's physiological data;
[0007] Input the features of the physiological data of each modality into the evidence neural network corresponding to each modality to obtain evidence e′ that the features of the physiological data of each modality are assigned to the kth fatigue level category k , k=1,…,K, K represents the number of fatigue degree categories;
[0008] Based on the evidence kAfter generating the opinions corresponding to each modality, the opinions corresponding to all modalities are fused to obtain the fused opinion w′;
[0009] The features of physiological data from each modality are fused to form the evidence e′ assigned to the kth fatigue level category k , and based on the obtained fusion evidence e″ k Calculate the probability p′ that the pilot is in the kth fatigue level category k ;
[0010] Based on the maximum probability p′ k A pilot fatigue level category is determined, and a confidence level of the pilot fatigue level category is determined based on the fused opinion w′.
[0011] In one embodiment of the present invention, the viewpoints corresponding to each modality include: the confidence level, prior probability, and unreliability of the features of the physiological data of each modality being assigned to the kth fatigue level category and being assigned to various fatigue level categories;
[0012] The features of physiological data from each modality are fused to form the evidence e′ assigned to the kth fatigue level category k , and based on the obtained fusion evidence e″ k Calculate the probability p′ that the pilot is in the kth fatigue level category k The steps include:
[0013] Calculate the mean of the evidence that the features of the physiological data of each modality are assigned to the kth fatigue level category to obtain the fusion evidence e″ k ;
[0014] Based on the convergent evidence k Calculate the Dirichlet distribution parameter α′ corresponding to the fusion mode k And Dirichlet intensity S′:
[0015] α′ k =e″ k +1;
[0016]
[0017] According to the Dirichlet distribution parameter α′ k and Dirichlet intensity S′, calculate the probability p′ that the pilot is in the kth fatigue level category k :
[0018] p′ k =α′ k / S′.
[0019] In one embodiment of the present invention, the multiple modalities include: EEG modality, image modality and ECG modality;
[0020] The steps of obtaining the physiological data of multiple modes of the pilot and preprocessing them to obtain the characteristics of the physiological data of each mode include:
[0021] Collecting the pilot's electroencephalogram (EEG) signal, and extracting the time domain features, frequency domain features, and time-frequency domain features of the EEG signal;
[0022] Collecting image data of the pilot, and extracting eye movement features and facial expression features of the pilot based on the image data;
[0023] The pilot's electrocardiogram (ECG) signal is collected, and the ECG features of the ECG signal are extracted.
[0024] In one embodiment of the present invention, the evidential neural network corresponding to each modality is pre-trained according to the following steps:
[0025] Acquire multiple physiological data samples of each modality and perform preprocessing to obtain features of the multiple physiological data samples of each modality;
[0026] Input the features of multiple physiological data samples of the vth modality into the evidence neural network to be trained, and collect evidence that the features of the nth physiological data sample of the vth modality are assigned to the kth fatigue level category v = 2, ..., V, V represents the number of modalities, n = 2, ..., N, N represents the number of physiological data samples of the vth modality;
[0027] According to the evidence and the Dirichlet distribution parameter corresponding to the feature of the nth physiological data sample of the vth modality Calculate the prior probability that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category Confidence and the degree of unreliability with which they are assigned to various fatigue levels
[0028] Based on the prior probability The confidence and the unreliability Generate an opinion for the nth physiological data sample under the vth modality;
[0029] The views from all modalities are integrated to obtain the integrated view;
[0030] Quantify the degree of conflict between views in different modalities;
[0031] Calculate the loss value based on the preset loss function, the quantitative results of the conflict degree and the fusion viewpoints;
[0032] Determine whether the loss value meets the preset conditions; if not, adjust the parameters of the evidence neural network to be trained corresponding to each modality through back propagation; if so, obtain the evidence neural network corresponding to the vth modality.
[0033] In one embodiment of the present invention, according to the evidence and the Dirichlet distribution parameter corresponding to the feature of the nth physiological data sample of the vth modality Calculate the prior probability that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category Confidence and the degree of unreliability with which they are assigned to various fatigue levels The steps include:
[0034] According to the evidence Calculate the Dirichlet intensity corresponding to the feature of the nth physiological data sample of the vth modality and the K-dimensional Dirichlet distribution parameter
[0035]
[0036] in, Denote the Dirichlet distribution parameters The 1st,…,Kth dimensions in , T represents transpose;
[0037] According to the Dirichlet intensity and the Dirichlet distribution parameter Calculate the confidence that the features of the nth physiological data sample input under the vth modality are assigned to the kth fatigue level category and the degree of unreliability with which they are assigned to various fatigue levels
[0038] Calculate the prior probability that the feature of the nth physiological data sample input under the vth modality is assigned to the kth fatigue level category in,
[0039] In one embodiment of the present invention, the generated opinion representation for the nth physiological data sample under the vth modality is:
[0040]
[0041] In the formula,
[0042] In one embodiment of the present invention, the confidence that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category is calculated according to the following formulas: and the degree of unreliability with which they are assigned to various fatigue levels
[0043]
[0044]
[0045] In one embodiment of the present invention, the step of fusing the viewpoints under all modalities to obtain a fused viewpoint includes:
[0046] The Dirichlet distribution parameters in the opinions under each modality and the features of the nth physiological data sample are assigned to the confidence and unreliability of the kth fatigue level category, and the fused opinion is obtained.
[0047] In one embodiment of the present invention, the viewpoints in all modes are fused according to the following formula:
[0048]
[0049] In the formula, They represent the opinions on the nth physiological data samples under the 1st, 2nd, …, Vth modalities, ◇ represents the fusion operation, and w represents the fusion opinion;
[0050] According to the following formula, the generated opinion of the nth physiological data sample under the i-th and j-th modalities is To perform the fusion:
[0051] In the formula, Indicates the confidence that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category The fusion result of Indicates the unreliability of the feature of the nth physiological data of the i-th and j-th modalities being assigned to the k-th fatigue level category The fusion result of The prior probability that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category fusion result.
[0052] In one embodiment of the present invention, the generated opinion of the nth physiological data sample under the i-th and j-th modalities is calculated according to the following formula: To quantify the degree of conflict:
[0053]
[0054] in, express and The projection distance between express and The conjunction certainty between them;
[0055] in,
[0056]
[0057]
[0058] in, They represent the probability that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category,
[0059] Compared with the prior art, the present invention has the following beneficial effects:
[0060] The present invention provides a pilot fatigue status monitoring method based on credible conflict multimodal learning. First, the evidence of the characteristics of the pilot's physiological data in each mode is obtained by using an evidential neural network to collect the confidence that the characteristics of the physiological data in each mode are assigned to each fatigue level category, thereby constructing the viewpoint corresponding to each mode, and finally determining the fatigue level category to which the pilot belongs based on the fusion viewpoint after all the viewpoints are fused. Compared with the existing multimodal learning method, the present invention can perceive the conflict between data specific to different modes because it aggregates the conflicting viewpoints, thereby showing excellent performance on conflicting multimodal data, and realizing the quantification of the conflict degree between modes and the credibility of decisions.
[0061] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 It is a flow chart of a pilot fatigue state monitoring method based on credible conflict multimodal learning provided by an embodiment of the present invention;
[0063] Figure 2 is a schematic diagram of a pilot fatigue state monitoring method based on credible conflict multimodal learning provided by an embodiment of the present invention;
[0064] Figure 3 is a visualization result diagram of the conflict degrees of different modes on a normal data set and a conflict data set provided by an embodiment of the present invention;
[0065] Figure 4This is a visualization result diagram of uncertainty density of normal and conflicting test sets provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0066] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.
[0067] Figure 1 1 is a flow chart of a pilot fatigue status monitoring method based on credible conflict multimodal learning provided by an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a pilot fatigue state monitoring method based on credible conflict multimodal learning, comprising:
[0068] S11, acquiring physiological data of multiple modes of the pilot and performing preprocessing to obtain characteristics of the physiological data of each mode;
[0069] S12, input the features of the physiological data of each modality into the evidence neural network corresponding to each modality, and obtain the evidence e′ that the features of the physiological data of each modality are assigned to the kth fatigue level category k , k=1,…,K, K represents the number of fatigue degree categories;
[0070] S13. Based on evidence e′ k After generating the viewpoints corresponding to each modality, the viewpoints corresponding to all modalities are fused to obtain the fused viewpoint w′ corresponding to each modality;
[0071] S14. Evidence e′ of the k-th fatigue level category assigned to the features of the physiological data of each modality k , and based on the obtained fusion evidence e″ k Calculate the probability p′ that the pilot is in the kth fatigue level category k ;
[0072] S15. Based on the maximum probability p′ k A pilot fatigue level category is determined, and a confidence level of the pilot fatigue level category is determined based on the fused viewpoint w′.
[0073] In this embodiment, the multiple modalities may include: EEG modality, image modality and ECG modality;
[0074] In step S11, the step of obtaining the physiological data of multiple modes of the pilot and preprocessing them to obtain the characteristics of the physiological data of each mode includes:
[0075] S1101, collecting the pilot's electroencephalogram (EEG) signal, and extracting the time domain features, frequency domain features, and time-frequency domain features of the EEG signal;
[0076] S1102, collecting image data of the pilot, and extracting eye movement features and facial expression features of the pilot based on the image data;
[0077] S1103. Collect the pilot's electrocardiogram signal and extract the electrocardiogram features of the electrocardiogram signal.
[0078] Specifically, this embodiment selects three modalities: EEG modality, image modality and ECG modality, and obtains the physiological data of the three modalities of the pilot, namely: the pilot's EEG signal, image data and ECG signal, wherein the image data can be an RGB image or an infrared image, and then uses the existing algorithm to extract the features of the physiological data of each modality. Exemplarily, the time domain features, frequency domain features and time-frequency domain features of the EEG signal are extracted by WPD (Wavelet Packet Decomposition), STFT (Short Time Fourier Transform) and KPCA (Kernel Principal Component Analysis); the eye feature vector is used to extract the eye movement feature, and the mouth feature vector is used to extract the expression feature through the facial feature point extraction algorithm based on Dlib; the ECG signal is converted into a spectrogram and a CQT (Constant Q Transform) spectrogram and the ECG feature is extracted.
[0079] Of course, in some other embodiments of the present application, physiological data of other modalities may also be collected, and the present application does not limit this.
[0080] In this embodiment, the evidence neural network corresponding to each modality is a DNN (Deep Neural Networks), which is pre-trained according to the following steps:
[0081] S21, acquiring multiple physiological data samples of each modality and performing preprocessing to obtain features of the multiple physiological data samples of each modality;
[0082] S22, inputting the features of the multiple physiological data samples of the vth modality into the evidence neural network to be trained, and collecting evidence that the features of the nth physiological data sample of the vth modality are assigned to the kth fatigue level category v = 1, ..., V, V represents the number of modalities, n = 1, ..., N, N represents the number of physiological data samples of the vth modality;
[0083] S23. Based on the evidence and the Dirichlet distribution parameter corresponding to the feature of the nth physiological data sample of the vth modality Calculate the prior probability that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category Confidence and the degree of unreliability with which they are assigned to various fatigue levels
[0084] S24. Based on prior probability Confidence and unreliability Generate an opinion for the nth physiological data sample under the vth modality;
[0085] S25, integrating the views under all modalities to obtain a fusion view;
[0086] S26, quantify the degree of conflict between opinions in different modes;
[0087] S27, calculating the loss value according to the preset loss function, the quantitative result of the conflict degree and the fusion viewpoint;
[0088] S28. Determine whether the loss value meets the preset conditions; if not, adjust the parameters of the evidence neural network to be trained corresponding to each modality through back propagation; if so, obtain the evidence neural network corresponding to the vth modality.
[0089] It should be understood that in the training process of the evidential neural network, multiple physiological data samples of EEG modality, image modality and ECG modality are first obtained, and the features of multiple physiological data samples of each modality are extracted through preprocessing.
[0090] Next, the features of multiple physiological data samples of the vth modality are input into the evidence neural network to be trained to collect evidence that the features of the nth physiological data sample of the vth modality are assigned to the kth fatigue level category. Among them, the Dirichlet probability density function (PDF) is used to form the category distribution, which can model the second-order uncertainty, while the probability value in the softmax layer only captures the first-order uncertainty.
[0091] The probability density function of the Dirichlet distribution is:
[0092]
[0093] In the formula, B(α) represents the K-dimensional polynomial function, represents the K-dimensional unit simplex, defined as:
[0094]
[0095] Optionally, in step S23, according to the evidence and the Dirichlet distribution parameter corresponding to the feature of the nth physiological data sample of the vth modality Calculate the prior probability that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category Confidence and the degree of unreliability with which they are assigned to various fatigue levels The steps include:
[0096] S2301, based on evidence Calculate the Dirichlet intensity corresponding to the feature of the nth physiological data sample of the vth modality and the K-dimensional Dirichlet distribution parameter
[0097]
[0098] in, Denote the Dirichlet distribution parameters The 1st,…,Kth dimensions in , T represents transpose;
[0099] S2302, according to Dirichlet intensity and the Dirichlet distribution parameter Calculate the confidence that the features of the nth physiological data sample input under the vth modality are assigned to the kth fatigue level category and the degree of unreliability with which they are assigned to various fatigue levels
[0100] S2302: Calculate the prior probability that the feature of the nth physiological data sample input under the vth mode is assigned to the kth fatigue level category in,
[0101] The Dirichlet probability density function naturally reflects the random sampling of statistical events, which is the basis for interpreting opinions as statistical measures of likelihood. In this embodiment, for the K-classification problem, the opinion generated for the nth physiological data under one modality can be represented as an ordered triplet consisting of confidence, unreliability, and prior probability.
[0102] Exemplarily, the generated opinion for the nth physiological data sample under the vth modality is expressed as:
[0103]
[0104] In the formula,
[0105] Optionally, the confidence that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category is calculated according to the following formulas: and the degree of unreliability with which they are assigned to various fatigue levels
[0106]
[0107]
[0108] In step S25, the step of fusing the viewpoints under all modalities to obtain a fused viewpoint includes:
[0109] The Dirichlet distribution parameters in the opinions under each modality and the features of the nth physiological data sample are assigned to the confidence and unreliability of the kth fatigue level category, and the fused opinion is obtained.
[0110] Specifically, the views under all modes are integrated according to the following formula:
[0111]
[0112] In the formula, They represent the opinions on the nth physiological data samples under the 1st, 2nd, …, Vth modalities, ◇ represents the fusion operation, and w represents the fusion opinion;
[0113] According to the following formula, the generated opinion of the nth physiological data sample under the i-th and j-th modalities is To perform the fusion:
[0114] In the formula, Indicates the confidence that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category The fusion result of Indicates the unreliability of the feature of the nth physiological data of the i-th and j-th modalities being assigned to the k-th fatigue level category The fusion result of The prior probability that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category fusion result.
[0115] In step S26, the generated viewpoints for the nth physiological data samples under the i-th and j-th modalities are calculated according to the following formula: Quantify the degree of conflict:
[0116]
[0117] in, express and The projection distance between Representation and The conjunction certainty between them;
[0118] in,
[0119]
[0120]
[0121] in, They represent the probability that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category,
[0122] In this embodiment, the preset loss function is constructed as:
[0123]
[0124] In the formula, α n,k represents the Dirichlet distribution parameter corresponding to the fusion view of the nth physiological data sample under all modalities, β and γ represent the hyperparameters of the dataset, where β represents the loss ratio of a single modality to the fusion modality. Since the data has been normalized in advance, β = 1, and γ represents the consistency loss ratio, which is empirically set to 1.
[0125] Generally speaking, those skilled in the art can convert a traditional DNN network structure into an evidence neural network by making adjustments based on the traditional DNN network structure. The adjustments mainly involve replacing the softmax layer with an activation layer (such as ReLU) and using the non-negative output of the layer as evidence, so that the Dirichlet distribution parameters can be obtained.
[0126] in,
[0127]
[0128] Where ψ(·) represents the digamma function.
[0129] However, this loss function does not guarantee that the evidence generated by incorrect labels will be lower. To solve this problem, an additional term, Kullback-Leibler (KL) divergence, is introduced into the preset loss function:
[0130]
[0131] Among them, D(p n |1) is a uniform Dirichlet distribution, For the nth physiological data sample, the predicted parameter α n is the parameter of the Dirichlet distribution after removing non-misleading evidence, and Γ(·) is the gamma function.
[0132] Therefore, for the nth physiological data sample, given the Dirichlet distribution parameter α n The loss is:
[0133] L acc (α n )=L ace (α n )+λ t L KL (α n ),
[0134] Among them, the annealing coefficient λ t =min(1.0,t / T)∈[0,1], t represents the current iteration number, and T represents the annealing step number.
[0135] This embodiment can prevent misclassified physiological data samples from converging to a uniform distribution too early by gradually increasing the influence of the KL divergence in the loss function.
[0136] In addition, in order to ensure the consistency of the results between different opinions during the training process, a method was adopted to minimize the degree of conflict between opinions. The consistency loss is calculated as follows:
[0137]
[0138] Similarly, similar to the training process, the viewpoint corresponding to each modality in step S13 may also include: the confidence, unreliability and prior probability of the features of the physiological data of each modality being assigned to the kth fatigue level category;
[0139] In step S14, the features of the physiological data of each modality are fused and assigned to the evidence e′ of the kth fatigue level category. k , and based on the obtained fusion evidence e″ k Calculate the probability p′ that the pilot is in the kth fatigue level category k The steps include:
[0140] Calculate the mean of the evidence that the features of the physiological data of each modality are assigned to the kth fatigue level category to obtain the fusion evidence e″ k ;
[0141] Based on the convergent evidence k Calculate the Dirichlet distribution parameter α′ corresponding to the fusion mode k And Dirichlet intensity S′:
[0142] α′ k =e″ k +1;
[0143]
[0144] According to the Dirichlet distribution parameter α′ k and Dirichlet intensity S′, calculate the probability p of the pilot belonging to the kth fatigue level category ' k :
[0145] p′ k =α′ k / S′.
[0146] Figure 3 It is a visualization result diagram of the conflict degree of different modalities on a normal data set and a conflict data set provided by an embodiment of the present invention, wherein the horizontal axis and the vertical axis respectively represent the conflict degree between the corresponding two modalities, and the six squares respectively represent the time domain perspective, frequency domain perspective and time-frequency domain perspective of the EEG modality, the eye movement perspective and expression perspective of the image modality, and the ECG modality.
[0147] Figure 4 : is a visualization result diagram of uncertainty density of normal and conflicting test sets provided by an embodiment of the present invention, where the horizontal axis is uncertainty and the vertical axis is density. Figure 4 Four sets of pilot physiological signal data sets are shown.
[0148] It can be seen from the above embodiments that the beneficial effects of the present invention are:
[0149] The present invention provides a pilot fatigue state monitoring method based on credible conflict multimodal learning. First, evidence neural network is used to obtain evidence of the characteristics of physiological data of the pilot in each mode, so as to collect the confidence of the characteristics of physiological data in each mode being assigned to each fatigue degree category, so as to construct the viewpoint corresponding to each mode, and finally determine the fatigue degree category to which the pilot belongs based on the fusion viewpoint after all the viewpoints are fused. Compared with the existing multimodal learning method, the present invention can perceive the conflict between data specific to different modes because it aggregates the conflicting viewpoints, thereby showing excellent performance on conflicting multimodal data, and realizing the quantification of the conflict degree between modes and the credibility of decision-making. In the description of the present invention, the description of reference terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. Furthermore, those skilled in the art may combine and couple different embodiments or examples described in this specification.
[0150] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A pilot fatigue status monitoring method based on credible conflict multimodal learning, It is characterized in that include: Acquire the pilot's physiological data in multiple modes and perform preprocessing to obtain the characteristics of each mode's physiological data; The features of the physiological data of each modality are input into the evidence neural network corresponding to each modality to obtain the evidence e that the features of the physiological data of each modality are assigned to the kth fatigue level category. ′ k , k=1,…,K, K represents the number of fatigue degree categories; Based on the evidence ′ k After generating the viewpoints corresponding to each modality, the viewpoints corresponding to all modalities are fused to obtain the fused viewpoint w ′ ; The features of physiological data from each modality are fused to assign evidence to the kth fatigue level category. ′ k , and based on the obtained fusion evidence e″ k Calculate the probability p′ that the pilot is in the kth fatigue level category k ; Based on the maximum probability p ′ k Determine the pilot's fatigue category and, based on the fusion viewpoint, ′ determining a confidence level for the pilot's fatigue level category; wherein, The evidence neural network corresponding to each modality is pre-trained according to the following steps: Acquire multiple physiological data samples of each modality and perform preprocessing to obtain features of the multiple physiological data samples of each modality; Input the features of multiple physiological data samples of the vth modality into the evidence neural network to be trained, and collect evidence that the features of the nth physiological data sample of the vth modality are assigned to the kth fatigue level category V represents the number of modalities, n = 1,…,N, N represents the number of physiological data samples of the vth modality; According to the evidence and the Dirichlet distribution parameter corresponding to the feature of the nth physiological data sample of the vth modality Calculate the prior probability that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category Confidence and the degree of unreliability with which they are assigned to various fatigue levels Based on the prior probability The confidence and the unreliability Generate an opinion for the nth physiological data sample under the vth modality; The views from all modalities are integrated to obtain the integrated view; Quantify the degree of conflict between views in different modalities; Calculate the loss value based on the preset loss function, the quantitative results of the conflict degree and the fusion viewpoints; Determine whether the loss value meets the preset conditions; if not, adjust the parameters of the evidence neural network to be trained corresponding to each modality through back propagation; if so, obtain the evidence neural network corresponding to the vth modality.
2. According to claim 1, the pilot fatigue state monitoring method based on credible conflict multimodal learning, It is characterized in that The viewpoints corresponding to each modality include: the confidence level, prior probability, and unreliability of the features of the physiological data of each modality being assigned to the kth fatigue level category; The features of physiological data from each modality are fused to assign evidence to the kth fatigue level category. ′ k , and based on the obtained fusion evidence e″ k Calculate the probability p′ that the pilot is in the kth fatigue level category k The steps include: Calculate the mean of the evidence that the features of the physiological data of each modality are assigned to the kth fatigue level category to obtain the fusion evidence e″ k ; Based on the convergent evidence k Calculate the Dirichlet distribution parameter α′ corresponding to the fusion mode k and Dirichlet intensity S ′ : α′ k =e″ k +1; According to the Dirichlet distribution parameter α ′ k and Dirichlet intensity S ′ , calculate the probability p that the pilot is in the kth fatigue level category ′ k : p ′ k =a ′ k / S ′ 。 3. The pilot fatigue status monitoring method based on credible conflict multimodal learning according to claim 2, It is characterized in that The multiple modalities include: EEG modality, image modality and ECG modality; The steps of obtaining the physiological data of multiple modes of the pilot and preprocessing them to obtain the characteristics of the physiological data of each mode include: Collecting the pilot's electroencephalogram (EEG) signal, and extracting the time domain features, frequency domain features, and time-frequency domain features of the EEG signal; Collecting image data of the pilot, and extracting eye movement features and facial expression features of the pilot based on the image data; The pilot's electrocardiogram (ECG) signal is collected, and the ECG features of the ECG signal are extracted.
4. The pilot fatigue status monitoring method based on credible conflict multimodal learning according to claim 1, It is characterized in that According to the evidence and the Dirichlet distribution parameter corresponding to the feature of the nth physiological data sample of the vth modality Calculate the prior probability that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category Confidence and the degree of unreliability with which they are assigned to various fatigue levels The steps include: According to the evidence Calculate the Dirichlet intensity corresponding to the feature of the nth physiological data sample of the vth modality and the K-dimensional Dirichlet distribution parameter in, Denote the Dirichlet distribution parameters The 1st,…,Kth dimensions in , T represents transpose; According to the Dirichlet intensity and the Dirichlet distribution parameter Calculate the confidence that the features of the nth physiological data sample input under the vth modality are assigned to the kth fatigue level category and the degree of unreliability with which they are assigned to various fatigue levels Calculate the prior probability that the feature of the nth physiological data sample input under the vth modality is assigned to the kth fatigue level category in, 5. The pilot fatigue status monitoring method based on credible conflict multimodal learning according to claim 4, It is characterized in that The generated opinion representation for the nth physiological data sample under the vth modality is: In the formula, 6. The pilot fatigue status monitoring method based on credible conflict multimodal learning according to claim 4, It is characterized in that The confidence that the feature of the nth physiological data sample of the vth modality is assigned to the kth fatigue level category is calculated according to the following formulas: and the degree of unreliability with which they are assigned to various fatigue levels 7. The pilot fatigue status monitoring method based on credible conflict multimodal learning according to claim 1, It is characterized in that The steps of integrating the views from all modalities to obtain the integrated view include: The Dirichlet distribution parameters in the opinions under each modality and the features of the nth physiological data sample are assigned to the confidence and unreliability of the kth fatigue level category, and the fused opinion is obtained.
8. The pilot fatigue status monitoring method based on credible conflict multimodal learning according to claim 7, It is characterized in that The views from all modes are integrated according to the following formula: In the formula, They represent the opinions on the nth physiological data samples under the 1st, 2nd, …, Vth modalities, ◇ represents the fusion operation, and w represents the fusion opinion; According to the following formula, the generated opinion of the nth physiological data sample under the i-th and j-th modalities is To perform the fusion: In the formula, Indicates the confidence that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category The fusion result of Indicates the unreliability of the feature of the nth physiological data of the i-th and j-th modalities being assigned to the k-th fatigue level category The fusion result of The prior probability that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category fusion result.
9. The pilot fatigue status monitoring method based on credible conflict multimodal learning according to claim 8, It is characterized in that According to the following formula, the generated opinion of the nth physiological data sample under the i-th and j-th modalities is To quantify the degree of conflict: in, express and The projection distance between express and The conjunction certainty between them; in, in, They represent the probability that the feature of the nth physiological data under the i-th and j-th modes is assigned to the k-th fatigue level category,
