Pilot attention state data acquisition and processing method, device and related equipment

Abstract

The application discloses a kind of pilot attention state data acquisition and processing method, device and related equipment, comprising: collecting original attention state data, original attention state data is time series data, original attention state data is carried out random mask operation, obtain first mask data, based on the time sequence correlation between adjacent channel signals, diagonal reconstruction is carried out to first mask data, obtain second mask data, first mask data and second mask data are input into fusion attention mechanism model, the dynamic weight between the two is analyzed by fusion attention mechanism model, obtain initial interpolation data, using the way of back propagation learning, initial interpolation data is used for the interpolation of real missing value, obtain complete interpolation data, based on original attention state data and complete interpolation data, obtain target attention state data.Using the present application can improve the accuracy of pilot attention state data acquisition and processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a pilot attention state data acquisition and processing method and device and related equipment. BACKGROUND

[0002] In the field of aviation, the state monitoring of pilots is crucial for flight safety. When performing flight tasks, pilots may be affected by various factors such as fatigue, stress, and distraction, which may lead to flight operation errors and thus threaten flight safety. Therefore, real-time monitoring of the physiological and psychological state of pilots in order to timely discover potential risks and take appropriate measures is an important part of aviation safety management.

[0003] In recent years, with the development of sensor technology and data acquisition technology, pilot state monitoring systems have gradually shifted from traditional subjective evaluation to objective monitoring based on physiological signals. These physiological signals include but are not limited to electroencephalogram (EEG), electrocardiogram (ECG), eye movement data, electromyogram (EMG), etc., which can reflect the pilot's attention, fatigue level, emotional state, and other key information.

[0004] Existing data missing processing methods are mainly divided into two categories: direct deletion and data interpolation. Direct deletion of missing data will result in incomplete data, introduce bias, and affect subsequent analysis results. Traditional interpolation methods, such as linear regression, mean / median interpolation, and k-nearest neighbors, are mostly based on strong assumptions about the data, which may introduce bias and perform poorly on small sample data. In recent years, deep learning-based time series interpolation methods have gradually emerged, such as BRITS based on recurrent neural networks (RNN) and NRTSI based on Transformer.

[0005] The inventors found that these methods perform well on large data sets, but still have shortcomings on small sample data, especially in the context of pilot state data with strong time correlation and sparse data, the collection and processing of these physiological signal data face many challenges:

[0006] Small amount of data and sparsity: Pilot state monitoring data usually belongs to small sample data. Due to the particularity of flight tasks and the complexity of data acquisition, the number of samples that can be obtained is limited. In addition, physiological signal data may have a large number of missing values on the time series, such as due to sensor failure, signal interference, or limitations of data acquisition equipment, resulting in incomplete data.

[0007] Poor data quality: Physiological signals are very weak and susceptible to electromagnetic noise, muscle activity, poor sensor contact, and other factors. These disturbances can cause noise, artifacts, or blank segments in the data, making the directly extracted feature information inaccurate, which in turn affects subsequent state assessment and risk warning.

[0008] Strong time correlation: The physiological signals of pilots have obvious time correlation. For example, there is a complex time-dependent relationship between different time steps of electroencephalogram data. Traditional data processing methods often ignore this time correlation, resulting in insufficient interpolation accuracy and poor data fitting effect.

[0009] High requirements for data integrity: The pilot state monitoring system has very high requirements for data integrity and accuracy.

[0010] Therefore, there is an urgent need for a method that can effectively handle the pilot state data missing problem. This method should make full use of the time correlation of the data, improve the interpolation accuracy of small sample data, and maintain good generalization ability in the case of limited data volume, thereby improving the accuracy of pilot attention state data acquisition and processing. SUMMARY

[0011] The embodiments of the present application provide a pilot attention state data acquisition and processing method, device, computer equipment and storage medium to improve the accuracy of pilot attention state data acquisition and processing.

[0012] To solve the above technical problems, the embodiments of the present application provide a pilot attention state data acquisition and processing method, comprising:

[0013] Acquiring original attention state data, the original attention state data being time series data;

[0014] Performing a random mask operation on the original attention state data to obtain first mask data;

[0015] Based on the time correlation between adjacent channel signals, performing diagonal reconstruction on the first mask data to obtain second mask data;

[0016] Inputting the first mask data and the second mask data into a fusion attention mechanism model, analyzing the dynamic weight between them through the fusion attention mechanism model, and obtaining initial interpolation data;

[0017] Using a backpropagation learning method, the initial interpolation data is used for interpolation of real missing values to obtain complete interpolation data;

[0018] Based on the original attention state data and the complete interpolation data, target attention state data is obtained.

[0019] Optionally, the diagonal reconstruction of the mask data based on the time sequence correlation between adjacent channel signals to obtain second mask data comprises:

[0020] The time correlation between the first mask data is strengthened through the multi-head attention mechanism layer.

[0021] The time correlation is combined, and the mask data is diagonally reconstructed to obtain complete time chain and time series data containing masks.

[0022] The time prediction reconstruction of the time series data containing masks is performed by using the gated recurrent unit to obtain the second mask data.

[0023] Optionally, the diagonal reconstruction of the mask data based on the time sequence correlation between adjacent channel signals to obtain complete time chain and time series data containing masks comprises:

[0024] The two diagonal time steps containing true values before and after the mask data of the adjacent two channels are analyzed.

[0025] The time sequence relationship between the time steps before and after is analyzed, and the mask data is reconstructed to obtain complete time chain and time series data containing masks.

[0026] Optionally, based on the original attention state data and the complete interpolation data, the target attention state data is obtained.

[0027] The original attention state data and the complete interpolation data are weighted by using a dynamic weight distribution function to obtain the target attention state data.

[0028] Optionally, the fusion attention mechanism model main body structure is a Transformer structure, and the Transformer structure comprises a position encoding and a feedforward network, and the position encoding is used to limit the sequence order.

[0029] To solve the above technical problems, the embodiment of the application further provides a pilot attention state data acquisition and processing device, comprising:

[0030] The data acquisition module is configured to acquire original attention state data, wherein the original attention state data is time series data.

[0031] The mask generation module is configured to perform random mask operation on the original attention state data to obtain first mask data.

[0032] The mask reconstruction module is configured to perform diagonal reconstruction on the first mask data based on the time sequence correlation between adjacent channel signals to obtain second mask data.

[0033] a data fusion module configured to input the first mask data and the second mask data into a fusion attention mechanism model, analyze dynamic weights between the first mask data and the second mask data by the fusion attention mechanism model, and obtain initial imputation data;

[0034] a back propagation module configured to use a back propagation learning method to use the initial imputation data to impute the real missing values, and obtain complete imputation data;

[0035] a state determination module configured to obtain target attention state data based on the original attention state data and the complete imputation data.

[0036] Optionally, the mask reconstruction module comprises:

[0037] a first association unit configured to strengthen time correlation between the first mask data by a multi-head attention mechanism layer;

[0038] a diagonal reconstruction unit configured to join the time correlation, perform adjacent channel diagonal reconstruction on the mask data, and obtain time series data of complete time chains and containing masks;

[0039] a second mask generation unit configured to use a gated recurrent unit to perform time prediction reconstruction on the time series data of complete time chains and containing masks, and obtain the second mask data.

[0040] Optionally, the diagonal reconstruction unit comprises:

[0041] a data analysis subunit configured to analyze two diagonal time steps containing real values before and after adjacent two channels of mask data;

[0042] a time sequence analysis subunit configured to analyze a time sequence relationship between the two diagonal time steps, reconstruct the mask data, and reconstruct the time series data of complete time chains and containing masks.

[0043] To solve the above technical problems, the embodiment of the application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the pilot attention state data acquisition and processing method.

[0044] To solve the above technical problems, the embodiment of the application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the pilot attention state data acquisition and processing method.

[0045] The present invention provides a method, apparatus, computer device, and storage medium for acquiring and processing pilot attention state data. The method acquires raw attention state data (time-series data), performs a random masking operation on the raw data to obtain a first mask, and then performs diagonal reconstruction based on the temporal correlation between adjacent channel signals to obtain a second mask. The first and second mask data are input into a fusion attention mechanism model, and the dynamic weights between them are analyzed to obtain initial imputation data. Using backpropagation learning, the initial imputation data is used to imputate the true missing values, resulting in complete imputation data. Based on the raw attention state data and the complete imputation data, the target attention state data is obtained. This method successfully solves the problems of insufficient imputation accuracy and data fitting caused by low-quality and insufficient quantity of small sample data by analyzing the correlation between the channels of the raw attention state data. Simultaneously, in the process of filling missing values, a diagonal reconstruction strategy based on strong temporal correlation between adjacent channels is introduced, significantly improving the imputation accuracy of the raw attention state data, thereby improving the accuracy of pilot attention state data acquisition and processing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0048] Figure 2 This is a flowchart of an embodiment of the pilot attention state data acquisition and processing method of this application;

[0049] Figure 3 This is a schematic diagram of one embodiment of the pilot attention state data acquisition and processing device according to this application;

[0050] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the use herein of terms such as "comprise" and "comprising", "have" and "having", "include" and "including" and "contain" and "containing" shall not be construed as indicating an exclusive inclusion, the use herein of terms such as "first", "second", and the like do not denote any order, quantity, or importance, but rather are used to nomenclature different elements.

[0052] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0054] Please refer to Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0055] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc.

[0056] The terminal devices 101, 102 and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers and desktop computers, and the like.

[0057] The server 105 can be a server providing various services, for example, a background server providing support for a page displayed on the terminal devices 101, 102 and 103.

[0058] It should be noted that the pilot attention state data acquisition and processing method provided by the embodiments of the present application is executed by the server, and accordingly, the pilot attention state data acquisition and processing device is arranged in the server.

[0059] It should be understood that, Figure 1 The number of terminal devices, networks and servers in

[0060] Please refer to Figure 2 , Figure 2 A pilot attention state data acquisition and processing method provided by an embodiment of the present application is shown, which is taken as an example for description of the server in Figure 1 The following is a detailed description:

[0061] S201: Collecting original attention state data, the original attention state data being time series data.

[0062] In particular, the present embodiment employs data from the Flight Crew Physiological Data dataset, which was collected to help detect the attentional state of pilots, as the primary raw attentional state data. This dataset captures physiological data from pilots / co-pilots entering and exiting a flight simulator. The benchmark test training set consists of a controlled experiment collected in a non-flying environment outside of the flight simulator. It consists of a full flight (takeoff, flight, and landing) in the flight simulator. The pilot experiences a distraction designed to induce one of the following three cognitive states: Channelized Attention (CA); Divided Attention (DA); Startle (SS). This test set contains a full flight simulation during which the pilot can experience any state (but not more than one at a time). Each sensor runs at a sampling rate of 256 Hz. Since this is physiological data from real people, there will be noise and artifacts in the data.

[0063] PhysioNet 2012 dataset as a generalization validation: The PhysioNet 2012 challenge dataset contains 12,000 multivariate clinical time series samples collected from intensive care unit (ICU) patients. Each sample records data for the first 48 hours of a patient's stay in the ICU. Depending on the patient's condition, up to 37 time series variables can be measured, such as body temperature, heart rate, blood pressure. Measurements can be collected at regular intervals (every hour or every day), or recorded at irregular intervals (only when needed). Not all samples contain all variables. This dataset is very sparse, with a total of 80% missing values. The following 12 samples are excluded because they contain no time series information at all: 147514, 142731, 145611, 140501, 155655, 143656, 156254, 150309, 140936, 141264, 150649, 142998.

[0064] Auditory evoked potential EEG-Biometric: The dataset contains 240 two-minute electroencephalogram recordings obtained from 20 volunteers. The data includes resting state and auditory stimulation experiments by analyzing the resting state, comparing the resting state with eyes closed and open, studying the effects of auditory stimulation on neuronal mechanisms after auditory stimulation, and analyzing how different languages affect human evoked potentials. The dataset measures four channels T7, F8, Cz, and P4 to collect electroencephalogram signals. Artificial masking of the dataset at 0.2 adds missing values.

[0065] S202: Randomly mask the original attentional state data to obtain first masked data.

[0066] Specifically, the original data is randomly masked to manufacture artificial masks, and the time correlation between the data signals is strengthened through a multi-head attention mechanism layer. The masks are reconstructed through adjacent diagonal channels to mine the time sequence clues between the adjacent diagonal channels, and the artificial masks are processed by a gated recurrent unit (GRU) for time prediction reconstruction. The artificial mask reconstruction Y value obtained by reconstruction and the true X value at the artificial mask in the original data enter a fusion attention mechanism layer, and the final interpolation value is obtained through learning by analyzing the dynamic weight between them. The function mechanism learned from the fusion attention mechanism layer is back-propagated, so that the real missing value is obtained through the fusion attention mechanism layer, and finally the complete interpolation data is obtained.

[0067] S203: Diagonal reconstruction is performed on the first mask data based on the time sequence correlation between adjacent channel signals to obtain second mask data.

[0068] In a specific optional implementation, the diagonal reconstruction is performed on the first mask data based on the time sequence correlation between adjacent channel signals to obtain second mask data, including:

[0069] The time correlation between the first mask data is strengthened through a multi-head attention mechanism layer;

[0070] The time sequence data containing the mask and the complete time chain are obtained by performing adjacent channel diagonal reconstruction on the mask data based on the time correlation;

[0071] The time sequence data containing the mask is time-predicted reconstructed by a gated recurrent unit to obtain second mask data.

[0072] In a specific optional implementation, the time sequence data containing the mask and the complete time chain are obtained by performing adjacent channel diagonal reconstruction on the mask data based on the time correlation, including:

[0073] The two diagonal time steps containing the true value before and after the mask data of the adjacent two channels are analyzed;

[0074] The time sequence relationship between the time steps before and after is analyzed, and the mask data is reconstructed to obtain the time sequence data containing the mask and the complete time chain.

[0075] Specifically, the time correlation of the joint is used to reconstruct the adjacent channel diagonal of the mask data, and the complete time chain and the time series data containing the mask are obtained. The present embodiment adopts a small sample data interpolation algorithm (Fusion Attention mechanism and adjacent reconstruction, FADR). After random artificial masking, the diagonal reconstruction is performed through the time correlation between adjacent channel signals, and the artificial mask of the two adjacent channels The two diagonals containing the true value before and after the value are , The time characteristics are analyzed, the time relationship between the time steps before and after is analyzed, the artificial mask The value is reconstructed to form a time series data similar to the complete time chain and containing the artificial mask. The gating cycle unit takes this virtual time series data as input data to predict and reconstruct the artificial mask position to obtain . The fusion attention mechanism layer is used to dynamically allocate weights to the original true data and the first reconstructed data to obtain the final prediction interpolation value of the second reconstruction and fill it into the data.

[0076] S204: input the first mask data and the second mask data into the fusion attention mechanism model, analyze the dynamic weight between the two through the fusion attention mechanism model, and obtain the initial interpolation data.

[0077] Optionally, the main structure of the fusion attention mechanism model is a Transformer structure, which includes position encoding and a feedforward network. The position encoding is used to limit the sequence order.

[0078] Specifically, attention mechanism is widely used in sequence modeling. Given a sequence, it is mapped to a query vector with a dimension of , a key vector with a dimension of and a value vector with a dimension of . The scaled dot product can effectively calculate the attention score (or attention map) between and . Then apply the softmax function to obtain the attention weight. The final output is the weighted . The whole process is shown in the following equation:

[0079]

[0080] In this algorithm, because there is artificial mask, the input value of the t step in the time series cannot use their own real value, that is, it is not allowed to use its own real value to make prediction. Therefore, their estimates only rely on the input values from (T-1) time step and (T+1) time step. Through the analysis of the virtual time series, the learned fusion attention weight distribution function , Pass into the next module.

[0081] In this way, the timing dependence and feature correlation between time steps in the electroencephalogram multi-channel data are captured. Subsequently, the fusion attention mechanism (Fusion Attention Mechanism) is defined as:

[0082]

[0083]

[0084] , and is the parameter of the linear layer that projects the input to , and respectively. is the parameter of the output layer.

[0085] It should be noted that in the Transformer structure of the present embodiment, position encoding is used for sequence ordering to capture the dependence of the data. In addition, a fully connected feedforward network is also applied after each attention layer. In FADR, part of the position encoding and feedforward network in the Transformer is applied.

[0086] The position encoding is composed of sine and cosine functions, as shown in the following equation, for the sake of brevity, use p to represent the position encoding, is the time step position, is the dimension,

[0087] ,

[0088] .

[0089] The feedforward network contains two linear transformations, between which there is an activation function, as follows:

[0090] .

[0091] Further, the dynamic optimization of the model is improved by the objective function Loss obtained in the training, as follows:

[0092]

[0093] wherein is the observation value of the first mask in the electroencephalogram data is the sample value reconstructed at the first mask in the original data domain; 、 is the weight parameter of the fusion attention mechanism layer; is the dynamic sliding window length; N is the total number of windows.

[0094] S205: The initial interpolation data is used for interpolation of the real missing values in a way of back propagation learning, to obtain complete interpolation data.

[0095] Specifically, the real missing values are processed by the fusion attention mechanism layer, integrated with the mask output data, and the optimal weight function is retained, which is reversely given to the original data to interpolate the real missing values, to obtain complete interpolation data.

[0096] S206: Based on the original attention state data and the complete interpolation data, target attention state data is obtained.

[0097] In a specific optional embodiment, based on the original attention state data and the complete interpolation data, target attention state data is obtained, including:

[0098] The original attention state data and the complete interpolation data are weighted and processed by a dynamic weight allocation function to obtain the target attention state data.

[0099] Further, three indicators are used to evaluate the interpolation performance of the method: MAE (mean absolute error), MSE (mean square error) and R 2 (mean relative error). Imputation is the interpolation value; real is the real target value:

[0100]

[0101]

[0102]

[0103] Since diagonal adjacent reconstruction is used, and the mask mode is random mask, it may cause some adjacent channel data values to change. It is set that if the real value change is within 0.1 percentage points, the change amount can be ignored, that is, it is considered that no change has occurred.

[0104] In this embodiment, five excellent time series prediction interpolation models are selected as the baseline. The specific information of the baseline method is as follows:

[0105] BRITS: Based on a recurrent neural network (RNN), it is used for missing value interpolation of time series data. Missing values can be learned directly in a bidirectional recurrent dynamic system. The interpolated values are treated as variables in the RNN and can be effectively updated during the backpropagation process, achieving interpolation of time series. Its performance is highly dependent on the quality of the input data. If there are a large number of noises or outliers in the time series data, the interpolation accuracy of the model may be affected.

[0106] M-RNN: Using the three-dimensional characteristics of data, through the interpolation block and interpolation block of the simultaneous multi-directional recurrent neural network, and can run across data streams, it realizes interpolation in data streams and estimation across data streams. It is more biased towards the combination of images and text. The feature representation of the two may not be completely aligned, which may limit the performance of the model. It inherits the structure of traditional RNN, making it difficult to effectively handle long-term dependencies in long sequences. Transformer: Completely based on attention mechanism to describe the global dependency between input and output, omitting the cycle and convolution, realizing parallelization. To achieve fast and efficient interpolation of time series data. Due to the lack of combination ability, the Transformer may generate false information that does not match the facts. SAITS: Through joint optimization method training, the missing values are learned by weighted combination of masking modules, capturing the temporal dependency and feature correlation between time steps. The model cannot obtain sufficient training when facing small sample data, which may lead to large error in the filled data.

[0107] GRU: is a variant derived from recurrent neural network, which solves the problem of long-term memory and gradient in backpropagation in traditional RNN through improved gate unit. However, due to the limitation of its gating structure, if it is not processed when facing large data sets, it may cause overfitting.

[0108] By comparing the experimental results in this embodiment with the experimental results of other five baseline methods, as shown in Table 1, the best experimental performance of the model on each data set is represented in bold font.

[0109] The indicators are sorted according to: mean absolute error (MAE) / mean square error (MSE) / coefficient of determination (R 2 ). MAE and MSE evaluation indicators: the lower the value, the better the performance. R 2 evaluation indicator: the higher the value, the better the model fits the data. From the table, it can be seen that: in the model, the evaluation indicators (MAE / MSE / R 2) to achieve the best: 0.135 / 0.246 / 0.912 and 0.124 / 0.143 / 0.947, respectively. The model SAITS is the best on PhysioNet20212, but the model can still maintain better performance data than other models. The model did not achieve the best on the PhysioNet20212 dataset, the main reason is that the total length of the sequence of the PhysioNet20212 dataset is too long and there are too many original missing values, which requires a lot of reconstruction tasks for the adjacent channel data, which makes it difficult for the adjacent channel to build enough relevance, resulting in unsatisfactory interpolation effect. Although the PhysioNet20212 dataset is difficult to interpolate, the model can still perform sub-optimal performance on large datasets.

[0110] The performance of the model on EEG-Biometric brain electrical physiological data with high correlation between channels is improved by nearly 3 percentage points and 1 percentage point in mean square error and coefficient of determination, respectively, compared to the sub-optimal model SAITS. Similarly, the performance of the model on Flight Crew Physiological brain electrical physiological data is improved by 7 percentage points and 4 percentage points in mean square error and coefficient of determination, respectively, compared to the sub-optimal model SAITS.

[0111] According to the performance indicators, the model can better process and fit the small sample physiological signal data with stronger correlation between channels, and achieve accurate interpolation of missing values. And on other larger data, it also maintains good generalization. The model GRU in the table does not have experimental results on the PhysioNet20212 dataset, because gradient explosion occurred during training.

[0112] Table 1 Comparison of performance indicators of different algorithms

[0113]

[0114] To further compare and verify the effectiveness of the adjacent channel diagonal reconstruction mechanism and the fusion attention mechanism in this paper, ablation experiments are performed on three datasets under the same experimental conditions, and the complete model without removing the two is compared. The adjacent channel diagonal reconstruction module is controlled to prove that the model has better performance on physiological signal data with small data volume but strong relationship between data. The fusion attention mechanism layer is controlled to prove that the setting of the fusion attention mechanism can better explain the relationship between the parameters of the data. In the model without fusion attention mechanism, the self-attention mechanism (SADR) of the SAITS model which performs better in different algorithm performance indicators is used to replace the experiment.

[0115] As shown in Table 2: first in the large amount of physiological signal data set on the experiment PhysioNet2012 data set, from the experimental results, it can be seen that FARD algorithm has obvious effect than other two groups of ablation model in large amount of data: the model of retaining adjacent diagonal reconstruction module is compared with the model of removing the basic model, which improves the performance index MAE of model reconstruction interpolation value by 2 percentage points, and has obvious advantage in data fitting degree, which shows that the data restoration degree is higher, and proves that the algorithm has more prominent optimization on the basis of ensuring data integrity; the model using fusion attention mechanism module is compared with the self-attention module, which has obvious advantage in data restoration degree, which improves nearly 8 percentage points. In the small sample EEG-Biometric and Flight Crew Physiological medical physiological data with strong correlation: the performance index MAE of the model retaining diagonal reconstruction is improved by 2 percentage points and 9 percentage points respectively compared with the model removing adjacent diagonal reconstruction module. And in the MSE performance index, the model retaining is improved by 7 percentage points and 12 percentage points respectively compared with the model removing adjacent diagonal reconstruction module, and the advantage in data fitting degree is more prominent; in the ablation experiment of fusion attention mechanism, FADR is compared with SADR, which improves the performance index EEG-Biometric and Flight Crew Physiological data set by 3 percentage points and 7 percentage points respectively in the MSE performance index, and by 1 percentage point and 6 percentage points respectively in the MAE performance index.

[0116] Table 2 Performance index comparison of ablation experiment

[0117]

[0118] In order to prove the rationality of FADR design, ablation experiment is carried out, and from the experimental results of ablation experiment, it is proved that the fusion attention mechanism module and the adjacent channel diagonal reconstruction module in FADR model play a key role in the model. The experimental results in the ablation experiment of the two modules show that whether it is a large amount of medical data set or a small sample of electroencephalogram data set. If only one module is used, the overall interpolation effect is not good, and even in some performance indicators, it is not as good as the effect of other models, which also reflects the rationality of the design of the fusion attention mechanism module and the adjacent channel diagonal reconstruction module in FADR model. And it shows that the model can effectively capture the time dependence between the data which is not easy to collect due to small sample size, so as to effectively interpolate the data.

[0119] In this embodiment, the original attention state data is collected, the original attention state data is time series data, random mask operation is performed on the original attention state data to obtain first mask data, diagonal reconstruction is performed on the first mask data based on the time sequence correlation between adjacent channel signals to obtain second mask data, the first mask data and the second mask data are input into a fusion attention mechanism model, the dynamic weight between the two is analyzed through the fusion attention mechanism model to obtain initial interpolation data, the initial interpolation data is used for interpolation of real missing values in a way of back propagation learning to obtain complete interpolation data, and target attention state data is obtained based on the original attention state data and the complete interpolation data. Through analyzing the correlation between channels of the original attention state data, the method successfully solves the problems of insufficient interpolation accuracy and insufficient data fitting caused by low quality and insufficient quantity of small sample data. At the same time, in the process of filling the missing values, the diagonal reconstruction strategy of strong time correlation between adjacent channels is introduced, which significantly improves the interpolation accuracy of the original attention state data, thereby improving the accuracy of pilot attention state data acquisition and processing.

[0120] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0121] Figure 3 The principle block diagram of the pilot attention state data acquisition and processing device corresponding to the pilot attention state data acquisition and processing method of the above embodiment is shown. As shown in Figure 3 The pilot attention state data acquisition and processing device includes a data acquisition module 31, a mask generation module 32, a mask reconstruction module 33, a data fusion module 34, a back propagation module 35 and a state determination module 36. The functions of each module are described in detail as follows:

[0122] The data acquisition module 31 is used for collecting original attention state data, and the original attention state data is time series data.

[0123] The mask generation module 32 is used for performing random mask operation on the original attention state data to obtain first mask data.

[0124] The mask reconstruction module 33 is used for performing diagonal reconstruction on the first mask data based on the time sequence correlation between adjacent channel signals to obtain second mask data.

[0125] The data fusion module 34 is used for inputting the first mask data and the second mask data into a fusion attention mechanism model, analyzing the dynamic weight between the two through the fusion attention mechanism model, and obtaining initial interpolation data.

[0126] Backpropagation module 35 is used to use backpropagation learning to impute the initial imputation data for the actual missing values, so as to obtain complete imputation data;

[0127] The state determination module 36 is used to obtain target attention state data based on the original attention state data and the complete interpolation data.

[0128] Optionally, the mask reconstruction module 33 includes:

[0129] The first association unit is used to enhance the temporal correlation between the first mask data through a multi-head attention mechanism layer;

[0130] The diagonal reconstruction unit is used to combine temporal correlations and perform diagonal reconstruction of adjacent channels on the mask data to obtain a complete time chain and time series data containing the mask.

[0131] The second mask generation unit is used to perform time prediction and reconstruction on time series data containing the mask using a gated loop unit to obtain the second mask data.

[0132] Optionally, the diagonal reconstruction unit includes:

[0133] The data analysis subunit is used to analyze the time steps containing the true values ​​at the two diagonal edges of the mask data of two adjacent channels.

[0134] The timing parsing subunit is used to parse the timing relationship between consecutive time steps, reconstruct the mask data, and reconstruct the complete time chain and time series data containing the mask.

[0135] Specific limitations regarding the pilot attention status data acquisition and processing device can be found in the limitations of the pilot attention status data acquisition and processing method described above, and will not be repeated here. Each module in the aforementioned pilot attention status data acquisition and processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the corresponding operations of each module.

[0136] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0137] The computer device 4 includes a memory 41, a processor 42, and a network interface 43, which are communicatively connected by a system bus. It should be noted that the computer device 4 is only shown with the components of the memory 41, the processor 42, and the network interface 43, but it should be understood that not all of the shown components are required to be implemented, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0138] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, and the like.

[0139] The memory 41 includes at least one type of readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or D interface display memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 41 can also include both the internal storage unit and the external storage device of the computer device 4. In the present embodiment, the memory 41 is generally used to store an operating system and various application software installed in the computer device 4, such as program codes of the pilot attention state data acquisition and processing method, and the like. In addition, the memory 41 can also be used to temporarily store various data that have been output or will be output.

[0140] The processor 42 may, in some embodiments, be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In the present embodiment, the processor 42 is configured to run program code or process data stored in the memory 41, such as program code of the pilot attention state data acquisition and processing method.

[0141] The network interface 43 may include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0142] The present application also provides another embodiment, i.e., to provide a computer readable storage medium, which stores an interface display program, the interface display program can be executed by at least one processor, so that the at least one processor executes the steps of the pilot attention state data acquisition and processing method as described above.

[0143] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0144] Obviously, the above-described embodiments are only some embodiments of the present application, not all embodiments, and the drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be realized in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some technical features. Any equivalent structure made by using the contents of the present application specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A method for collecting pilot attention state data features, characterized by, The method comprises the following steps: Collecting original attention state data, which is time series data; Randomly masking the original attention state data to obtain first masked data; Based on the time sequence correlation between adjacent channel signals, the first masked data is diagonally reconstructed to obtain second masked data; Input the first masked data and the second masked data into the fusion attention mechanism model, and allocate dynamic weights between them through the fusion attention mechanism model to obtain initial interpolation data; Using the back propagation learning method, the initial interpolation data is used for interpolation of real missing values to obtain complete interpolation data; Based on the original attention state data and the complete interpolation data, target attention state data is obtained. In the attention mechanism model, a fully connected feedforward network is applied after each attention layer, only using position encoding in the Transformer, and the first masked data and the second masked data are input into the fusion attention mechanism model, and the dynamic weights between them are allocated through the fusion attention mechanism model to obtain initial interpolation data, which includes: The main structure of the fusion attention mechanism model is a Transformer structure, which includes position encoding and a feedforward network, and the position encoding is used to define the sequence order. For the first mask data and the second mask data, map them to a dimension of query vector Dimensions key vector and dimension are value vector Scaling dot product calculation and Attention scores between; The attention weight is obtained by using a softmax function, and the final output is the weighted initial interpolation data and is expressed by the following equation: The input value at the t-th step in the time series is determined based on the input values from the (T-1)th step and the (T+1)th step, and the learned fusion attention weight distribution function is passed into the next module through the analysis of the virtual time series. In this way, the time step dependence and feature correlation in the electroencephalogram multi-channel data are captured.

2. The method of claim 1, wherein the pilot attention state data feature is collected by: The first masked data is reconstructed diagonally based on the time sequence correlation between adjacent channel signals to obtain second masked data, which includes: Through the multi-head attention mechanism layer, the time correlation between the first masked data is strengthened; The time correlation is combined to reconstruct the adjacent channel diagonal of the masked data to obtain complete time chain and time series data containing masks; The time prediction reconstruction of the time series data containing masks is performed by using the gated recurrent unit to obtain the second masked data.

3. The method of claim 2, wherein the pilot attention state data feature is collected by: The time correlation is combined to reconstruct the adjacent channel diagonal of the masked data to obtain complete time chain and time series data containing masks, which includes: Analyze the two diagonal time steps containing real values before and after the masked data of the adjacent two channels; Analyze the time sequence relationship between the front and rear time steps, and reconstruct the complete time chain and time series data containing masks by reconstructing the masked data.

4. The method of claim 1, wherein the pilot attention state data feature is collected by: Based on the original attention state data and the complete interpolation data, target attention state data is obtained, which includes: The original attention state data and the complete interpolation data are weighted by a dynamic weight distribution function to obtain the target attention state data.

5. The method of claim 1, wherein: The main structure of the fusion attention mechanism model is a Transformer structure, which includes position encoding and a feedforward network, and the position encoding is used to define the sequence order.

6. A pilot attention state data feature collection device, characterized by, The method comprises the following steps: The data acquisition module is configured to acquire original attention state data, which is time series data. The mask generation module is configured to perform random mask operation on the original attention state data to obtain first mask data. The mask reconstruction module is configured to perform diagonal reconstruction on the first mask data based on time sequence correlation between adjacent channel signals to obtain second mask data. The data fusion module is configured to input the first mask data and the second mask data into a fusion attention mechanism model, assign a dynamic weight between the first mask data and the second mask data through the fusion attention mechanism model, and obtain initial interpolation data. The main structure of the fusion attention mechanism model is a Transformer structure, which includes position encoding and a feedforward network. For the first mask data and the second mask data, map them to a dimension of query vector Dimensions key vector and dimension are value vector Scaling dot product calculation and Attention scores between; The attention weight is obtained by using a softmax function, and the final output is the weighted initial interpolation data and is expressed by the following equation: The input value at the t-th step in the time series is determined based on the input values from the (T-1)-th step and the (T+1)-th step. The learned fusion attention weight distribution function is passed into the next module through analysis of the virtual time series. The reverse propagation module is configured to use a backpropagation learning method to use the initial interpolation data for interpolation of real missing values to obtain complete interpolation data.

7. The pilot attention state data feature collection apparatus of claim 6, wherein The state determination module is configured to obtain target attention state data based on the original attention state data and the complete interpolation data. The mask reconstruction module includes: The first correlation unit is configured to strengthen the time correlation between the first mask data through a multi-head attention mechanism layer. The diagonal reconstruction unit is configured to join the time correlation and perform adjacent channel diagonal reconstruction on the mask data to obtain time series data with complete time chains and masks.

8. The pilot attention state data feature collection apparatus of claim 7, wherein, The second mask generation unit is configured to use a gated recurrent unit to perform time prediction reconstruction on the time series data with masks to obtain second mask data. The diagonal reconstruction unit includes: The data analysis subunit is configured to analyze two adjacent diagonal time steps containing real values before and after the mask data of two channels.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The time sequence analysis subunit is configured to analyze the time sequence relationship between the two time steps before and after the mask data, reconstruct the mask data, and reconstruct time series data with complete time chains and masks.

10. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The processor executes the computer program to implement the pilot attention state data feature acquisition method of any one of claims 1-5. The computer program is executed by the processor to implement the pilot attention state data feature acquisition method of any one of claims 1-5.

Images