Method and device for sensing controller working mode, electronic device, and storage medium

Abstract

The present invention discloses a method and device for perceiving a controller's working mode, an electronic device, and a storage medium, including: collecting eye movement data; preprocessing the eye movement data; preliminarily processing the preprocessed eye movement data to obtain eye movement features; classifying the eye movement features according to whether they are time-series related; further processing different types of eye movement features and extracting deep features respectively; splicing the deep features; and predicting the controller's working mode based on the spliced ​​deep features. The technical solution of the present invention can stably, accurately and effectively perceive and predict the controller's working state.

Description

Technical Field

[0001] The present invention belongs to the field of eye movement technology, and in particular relates to a method and device for perceiving a controller's working mode based on eye movement information, an electronic device, and a storage medium. Background Art

[0002] At present, most of the unsafe events in civil air traffic control are related to control errors, among which the failure of the controller's working mode is the main reason. This common phenomenon in the radar monitoring process has brought great safety hazards to the safe operation of air traffic control systems at home and abroad. Vision is the main way for controllers to monitor the status of aircraft, and the appropriate selection and processing of visual information is an indispensable cognitive function of controllers. However, judging and predicting the state of controllers only by the shallow basic features and a small number of partial features of the controller's eye movement information is likely to cause the deep characteristics hidden in the original data to be omitted, so it is inevitable that there is a lack of rigor in terms of accuracy and rationality. With the development of deep learning and machine learning, some researchers have begun to use methods such as CNN convolutional neural network and RNN recurrent neural network to identify the working mode of operators. Although some scholars have begun to use neural networks to process eye movement signals in recent years, the research on the application of neural networks and deep learning methods to process and identify the eye movement signals of controllers is still not in-depth enough, and the recognition performance needs to be improved. In addition, in existing research, the temporal characteristics of controllers' eye movement signals have not been fully explored. This patent develops a perception system for the controller's working mode in response to these current situations. It can effectively identify the controller's working mode, prevent the controller's working mode from failing, and improve the safety level of air traffic control system operation. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method and device, an electronic device, and a storage medium for perceiving a controller's working mode based on eye movement information, which can accurately identify the controller's working mode.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] The present invention provides a method for perceiving a controller's working mode based on eye movement information, comprising the following steps:

[0006] Step S1, collecting eye movement data;

[0007] Step S2, preprocessing the eye movement data;

[0008] Step S3, preliminarily processing the preprocessed eye movement data to obtain basic eye movement features;

[0009] Step S4, classifying basic eye movement features;

[0010] Step S5, extracting deep features from different types of basic eye movement features;

[0011] Step S6, splicing the deep features;

[0012] Step S7: predicting controlled forgetting based on the concatenated deep features.

[0013] Preferably, the eye movement data in step S1 is obtained by a head-mounted eye tracker.

[0014] Preferably, in step S2, the data preprocessing adopts a linear interpolation method to fill in the pupil diameter data lost due to blinking, and uses the average value of the data before and after the missing value to fill in the missing value caused by improper collection.

[0015] Preferably, the basic eye movement feature classification in step S4 is to classify the basic eye movement features into two types: timing-related basic eye movement features and timing-independent basic eye movement features according to whether they are timing-related.

[0016] Preferably, in step S5, a CNN-LSTM neural network is used to process the timing-related basic eye movement features to extract deep features; and a CNN neural network is used to process the timing-independent basic eye movement features to extract deep features.

[0017] Preferably, in step S6, the deep features obtained by processing the timing-related basic eye movement features through CNN-LSTM and the deep features obtained by processing the timing-independent basic eye movement features through CNN neural network are concatenated.

[0018] The present invention also provides a device for sensing a controller's working mode, comprising:

[0019] A collection module, used for collecting eye movement data;

[0020] A preprocessing module, used for preprocessing the eye movement data;

[0021] A preliminary processing module is used to perform preliminary processing on the preprocessed eye movement data to obtain basic eye movement features;

[0022] A classification module, used for classifying the basic eye movement features;

[0023] Extraction module, used to extract deep features from different types of basic eye movement features;

[0024] A splicing module, used for splicing the deep features;

[0025] The prediction module is used to predict controlled forgetting based on the concatenated deep features.

[0026] Preferably, the extraction module uses a CNN-LSTM neural network to process timing-related basic eye movement features and extract deep features; and simultaneously uses a CNN neural network to process timing-independent basic eye movement features and extract deep features.

[0027] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement a method for perceiving a controller's working mode.

[0028] The present invention also provides a storage medium, which stores machine executable instructions. When the machine executable instructions are called and executed by a processor, the machine executable instructions can prompt the processor to implement a method for perceiving a rights controller's working mode.

[0029] The present invention discloses the following technical effects:

[0030] Compared with the prior art, the advantage of the present invention is that it effectively improves the accuracy of the perception of the controller's working mode through the CNN-LSTM combined model and deep learning, and provides a certain reference for the development and research of future control warning work. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 is a flow chart of the method of the present invention;

[0033] Figure 2 Schematic diagram of the collection of eye movement features;

[0034] Figure 3 Schematic diagram of basic eye movement characteristics related to timing;

[0035] Figure 4 Schematic diagram of basic eye movement features that are independent of timing;

[0036] Figure 5 Extract schematic diagram for CNN;

[0037] Figure 6 Extract the schematic for the LSTM network. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Embodiment 1:

[0041] like Figure 1 As shown, the present invention provides a method for perceiving a controller's working mode based on eye movement information and a CNN-LSTM neural network, comprising the following steps:

[0042] The first step is to obtain eye movement data in .XLSX format using a head-mounted eye tracker.

[0043] In the second step, the original eye movement data is preprocessed, and the pupil diameter data lost due to blinking is filled in by linear interpolation. The missing values ​​caused by improper collection are filled in by the average value of the data before and after the missing values. After the original eye movement data is processed, it is truncated by sliding window, with a window length of 500 sampling points (5s), and adjacent windows overlap by 100 sampling points. The eye movement characteristics and control behaviors of the subjects within every 500 sampling points (5s) are regarded as an event. Figure 2 shown.

[0044] The third step is to process the eye movement data through relevant software to obtain 44 basic eye movement features and save them as .TXT format files.

[0045] The fourth step is to divide the basic eye movement features into two types according to whether they are time-related or not, namely, time-related basic eye movement features and time-independent basic eye movement features. There are 23 types of time-related basic eye movement features, such as Figure 3 As shown in Figure 2, there are 21 basic eye movement features that are independent of timing, such as Figure 4 As shown. The basic eye movement feature data related to the time sequence is recorded as X tc ; The basic eye movement feature data that is independent of time sequence is recorded as X tic .

[0046] The fifth step is to use the CNN-LSTM neural network to process the timing-related basic eye movement features and extract deep features. At the same time, the CNN neural network is used to process the timing-independent basic eye movement features and extract deep features.

[0047] When using the CNN-LSTM combined model to process the time-series related basic eye movement features, X tc As the lowest level input of the network, CNN is used to extract more advanced and abstract features. As shown in Figure (5), the CNN part is a two-dimensional convolution, and each event has 23 features.

[0048] Take every 6 events as a whole to form a 6×23 event sequence matrix. When performing CNN processing, the training batch is 64. Therefore, the input data dimension of CNN here is 64×6×23. 64 convolution kernels are defined in this method, so 64 different features can be extracted in the first layer of the network. Therefore, the output feature size of CNN here is 64×6×64. These three numbers represent the training batch size, event sequence length, and 64 high-level features. The feature data obtained after the CNN neural network is recorded as X deeper . ReLU is used as the activation function after the convolution operation.

[0049] Will Taking the i-th sample as input, the process can be expressed by the following formula:

[0050]

[0051]

[0052] Where C(·) is a function for extracting high-level features. Conv represents the convolution operation, and W refers to the parameters in the convolution calculation. It is the deep feature extracted from the i-th sample.

[0053] In order to increase the sample size and prevent the problem of model overfitting during training, the input data of the LSTM neural network is segmented and cut into 6 groups according to the event sequence dimension. The segmented data is sent to the LSTM network, which contains 3 hidden layers and 128 hidden layer units. Therefore, after being processed by the LSTM network, a time series feature output with a size of 64×6×128 is obtained. The output data is recorded as X tc-feature .Will As the input of the LSTM neural network, after processing, the time series feature output X is obtained tc-feature , which can be expressed by the following formula:

[0054]

[0055] L(·)=σ(ω0x t +U0h t -1+b0) (4)

[0056] Among them, σ is the activation function Sigmoid, ω0 and U0 are weight matrices, and b0 is the bias.

[0057] The other parameters in the formula are as follows:

[0058] h t =L(·)Tanh(c t ) (5)

[0059] Among them, c t is the current cell state, and its cell state at the previous moment is c t-1 The relationship between the two is

[0060] c t =f t c t-1 +i t c′ t (6)

[0061] Among them, c′ t is the current status of the unit,

[0062] c′ t =Tanh(ω c x t +U c h t-1 ) (7)

[0063] Among them, ω c and U c is the weight matrix.

[0064] i t =σ(ω i x t +U i h t -1+b t ) (8)

[0065] Among them, σ is the activation function Sigmoid ω i and U i is the weight matrix, b t For bias.

[0066] f t =σ(ω f x t +U f h t-1 +b f ) (9)

[0067] Among them, σ is the activation function Sigmoid, ω f and U f is the weight matrix, b f For bias.

[0068] Finally, the temporal feature output of the LSTM network is input into a fully connected layer to obtain a vector of length 2, denoted as X tc-output , as shown in Figure (6). The output after the fully connected layer can represent the overall characteristics of the time-series related eye movement data, which is also the total output of the CNN-LSTM module.

[0069] For the time-independent eye movement data, it is directly processed by the CNN network, and the processing flow is the same as the above CNN module. The output of the time-independent eye movement data after passing through the CNN module is recorded as X tic-output

[0070] Step 6: Process the time-series related eye movement features through CNN-LSTM to obtain the final deep feature X tc-output The final deep feature X obtained after the eye movement features are processed by the CNN neural network and are not related to the timing tic-output Splice and record the spliced ​​data as x final .

[0071] Step 7: The concatenated data x in step 6 final The data is sent to a classifier containing a fully connected layer for final classification, and the Softmax function is used to obtain a probabilistic prediction result, resulting in two probability values, which respectively represent the probability of whether the controller may forget control at the current moment.

[0072] Will As the i-th sample, y i To predict the results, this process can be expressed as follows:

[0073]

[0074]

[0075] A total of 100 epochs were trained, and the cross entropy loss function was used as the network loss function. The loss function is:

[0076]

[0077] Among them, y represents the category probability predicted by the model, class represents the correct category, and y[i] represents the predicted probability of the i-th category. This paper uses Adam as the optimizer when training the model, with an initial learning rate of 0.0001 and the Kaiming initialization method to initialize the network.

[0078] The method for perceiving the working mode of the controller of the present invention is to divide the basic eye movement features into two types: time-series related and time-series independent, and to perceive the working mode of the controller through the CNN-LSTM neural network model. The method can extract deeper features from the basic eye movement features, so compared with the method of only relying on basic eye movement indicators to identify the working mode of the controller, the recognition accuracy of this method is higher. In addition, since the eye movement signal has a certain continuity in time and space, and the CNN part can extract the spatiotemporal correlation between features to a certain extent, and the LSTM part that is good at processing time series has a better processing effect on the time-series related eye movement features, after combining the two, the spatiotemporal correlation between the eye movement features can be better extracted, and a more ideal recognition effect can be achieved.

[0079] Embodiment 2:

[0080] The present invention also provides a device for sensing a controller's working mode, comprising:

[0081] A collection module, used for collecting eye movement data;

[0082] A preprocessing module, used for preprocessing the eye movement data;

[0083] A preliminary processing module is used to perform preliminary processing on the preprocessed eye movement data to obtain basic eye movement features;

[0084] A classification module, used for classifying the basic eye movement features;

[0085] Extraction module, used to extract deep features from different types of basic eye movement features;

[0086] A splicing module, used for splicing the deep features;

[0087] The prediction module is used to predict controlled forgetting based on the concatenated deep features.

[0088] Preferably, the extraction module uses a CNN-LSTM neural network to process timing-related basic eye movement features and extract deep features; and simultaneously uses a CNN neural network to process timing-independent basic eye movement features and extract deep features.

[0089] The present invention effectively improves the accuracy of identifying the controller's working mode through the CNN-LSTM combined model and deep learning.

[0090] Embodiment 3:

[0091] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement a method for perceiving a controller's working mode.

[0092] The present invention also provides a storage medium, which stores machine executable instructions. When the machine executable instructions are called and executed by a processor, the machine executable instructions can prompt the processor to implement a method for perceiving a rights controller's working mode.

[0093] The embodiments described above are only descriptions of the preferred modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for sensing a controller's working mode, characterized in that: The following steps are involved: Step S1, collecting eye movement data; Step S2, preprocessing the eye movement data; Step S3, preliminarily processing the preprocessed eye movement data to obtain basic eye movement features; Step S4, classifying the basic eye movement features; Step S5, extracting deep features from different types of basic eye movement features; Step S6, splicing the deep features; Step S7, predicting controlled forgetting based on the spliced ​​deep features; Wherein, in the step S5, the CNN-LSTM neural network is used to process the timing-related basic eye movement features to extract the deep features; and the CNN neural network is used to process the timing-independent basic eye movement features to extract the deep features; In the step S6, the deep features obtained by processing the timing-related basic eye movement features through CNN-LSTM and the deep features obtained by processing the timing-independent basic eye movement features through CNN neural network are spliced; In step S7, the concatenated data x final The data is sent to a classifier containing a fully connected layer for final classification, and the Softmax function is used to obtain a probabilistic prediction result, resulting in two probability values, which respectively represent the probability of whether the controller may forget control at the current moment.

2. The method for sensing the working mode of a controller according to claim 1, characterized in that: The eye movement data in step S1 is obtained by a head-mounted eye tracker.

3. The method for sensing the working mode of a controller according to claim 1, characterized in that: The data preprocessing in step S2 uses a linear interpolation method to fill in the pupil diameter data lost due to blinking, and uses the average value of the data before and after the missing value to fill in the missing value caused by improper collection.

4. The method for sensing the working mode of a controller according to claim 1, characterized in that: In step S4, the basic eye movement features are classified into two types, namely, timing-related basic eye movement features and timing-independent basic eye movement features, according to whether they are timing-related or not.

5. A device for sensing a controller's working mode that implements the method for sensing a controller's working mode according to any one of claims 1 to 4, characterized in that: include: A collection module, used for collecting eye movement data; A preprocessing module, used for preprocessing the eye movement data; A preliminary processing module is used to perform preliminary processing on the preprocessed eye movement data to obtain basic eye movement features; A classification module, used for classifying the basic eye movement features; Extraction module, used to extract deep features from different types of basic eye movement features; A splicing module, used for splicing the deep features; The prediction module is used to predict controlled forgetting based on the concatenated deep features.

6. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method for perceiving the controller's working mode as described in any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium stores machine executable instructions, and when the machine executable instructions are called and executed by the processor, the machine executable instructions prompt the processor to implement the method for perceiving the controller's working mode as described in any one of claims 1 to 4.

Images