A cockpit HUD display control method, system and device

Abstract

The present invention provides a cockpit HUD display control method, system and device. In the original state, only the icons or name information of the flight parameter indicators are displayed on the flight cockpit HUD. The icons or names in each interface interest area are displayed according to different flashing frequencies. By collecting the pilot's electroencephalogram signal and eye movement signal, a multivariate eigenvalue is constructed, and based on the pre-trained recurrent neural network, it is judged whether the pilot is paying attention to a certain target according to the multivariate eigenvalue. On the basis that the pilot pays attention to the certain target, the target interface interest area of ​​interest is judged according to the coordinates of the gaze point, and then the corresponding reference signal is calculated according to the stimulation frequency of each icon or name information in the target interface interest area, and the target reference signal with the highest correlation with the electroencephalogram (EEG) signal is identified, and the target icon or target name information corresponding to the target reference signal is obtained, and its parameters are displayed. The information that the pilot is concerned about is actively identified and displayed, the pilot's observation pressure is reduced, and the efficiency of information acquisition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic equipment control, and in particular to a cockpit HUD display control method, system and device. Background Art

[0002] A head-up display (HUD) is usually installed above the central instrument panel of an aircraft cockpit. It can project various information such as status parameters, attack indicators, flight data, etc. in a unified and easily recognizable manner such as icons or characters onto the transparent combination glass directly in front of the pilot's line of sight, thereby realizing real-time information exchange and sharing. It is an important reference for pilots to obtain information and make judgments and decisions.

[0003] With the widespread popularization of new aviation technologies today, the research and development of various advanced airborne missiles and attack weapons, and the improvement of aircraft automation levels, HUD has been widely used. Moreover, with the development of modern aviation technology and the continuous advancement of informatization, the application of HUD has brought a new technological revolution to flight operations. At the same time, flight missions have gradually increased, and more and more information needs to be displayed on the aircraft interface. For example, flight altitude, speed, heading, target distance, weapon status and other information all need to be displayed on the HUD, and various contradictions have arisen between the interface and the user. First of all, the pilot needs to obtain more information in real time, and needs to pay attention to changes in various signals at all times and make corresponding reactions and decisions. With the upgrade of aviation systems, the use of a large number of computer display and control technologies and various new aviation equipment has put forward higher requirements on the overall quality and ability of pilots, and a large amount of information needs to be processed. Secondly, in different flight missions such as take-off / landing, navigation, air-to-air attack, and air-to-ground attack of new fighter jets, the information required and the focus of pilots are different. For example, the information area paid attention to in the take-off phase is not taken seriously in the combat phase, and the area paid attention to in the air combat phase is not prominent enough. Therefore, how to effectively and reasonably provide the information required for flight and improve the prominence of key information is directly related to the efficiency of the mission and even the success or failure. In addition, pilots have a certain subjective consciousness when screening information, and the information required for flight missions is not exactly the same. Therefore, how to accurately display the information that pilots want to pay attention to under different combat missions is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a cockpit HUD display control method, system and device to eliminate or improve one or more defects existing in the prior art and solve the problem that the existing flight cockpit HUD display content is complex and cannot highlight the information that the pilot is concerned about.

[0005] In one aspect, the present invention provides a cockpit HUD display control method based on human intelligence, the method comprising the following steps:

[0006] Displaying icons or names of multiple parameter indicators on the cockpit HUD, wherein the cockpit HUD presets multiple interface interest areas, and the icons or names in each interface interest area are displayed according to different flashing frequencies;

[0007] Acquiring EEG data and eye movement data of the operator in the cockpit at set time intervals;

[0008] Using the EEG data and the eye movement data as input, a pre-trained neural network model is used to determine whether the operator has an icon or name of interest;

[0009] If it is determined that the operator has an icon or name of interest, the coordinates of the gaze point in the current time interval are obtained according to the eye movement data, and the target interface interest area of ​​interest of the operator is determined according to the coordinate range of each interface interest area;

[0010] Obtaining the stimulation frequency of each icon or name in the target interface interest area, and obtaining a reference signal corresponding to each stimulation frequency;

[0011] Calculate the correlation coefficient between each reference signal and the EEG data, obtain the target reference signal with the largest correlation coefficient with the EEG data, and use the icon or name corresponding to the target reference signal as the target icon or target name;

[0012] The specific parameter information of the target icon or the target name is displayed on the cockpit HUD.

[0013] In some embodiments, the eye movement data includes: pupil diameter, gaze point coordinates, gaze time, and saccade amplitude;

[0014] The method further comprises: analyzing the EEG data to obtain the average power A of the delta wave at each set time interval δ 、θ wave average power A θ , α wave average power A α , β wave average power A β , γ wave average power A γ , and calculate (A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β as eigenvalues;

[0015] The pupil diameter, the fixation point coordinates, the fixation time, the eye saccade amplitude, and the A corresponding to the current time interval δ , said A θ , said A α , said A β , said A γ ,(A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β As the input of the neural network model.

[0016] In some embodiments, the icons or names in each interface interest area are displayed at different flashing frequencies, where the flashing frequency is a multiple of the cockpit HUD display refresh frequency.

[0017] In some embodiments, in the eye movement data, eye movement points with an eye movement speed threshold greater than 30° / s are marked as scan points, and eye movement points with an eye movement speed threshold less than or equal to 30° / s are marked as fixation points.

[0018] In some embodiments, after acquiring the EEG data and eye movement data of the operator in the cockpit at a set time interval, the method further includes:

[0019] Performing noise reduction preprocessing on the eye movement data;

[0020] The EEG data was subjected to 50 Hz notch filtering, down-sampled to 300 Hz, and then band-pass filtered to retain 12-110 Hz.

[0021] In some embodiments, the neural network model is a recurrent neural network model, and the training steps of the recurrent neural network model include:

[0022] Acquire a training sample set, wherein the training sample set includes multiple samples, each sample includes multi-channel sample EEG data and sample eye movement data of an operator within a specified time period, the sample EEG data includes a sample EEG signal, and the sample eye movement data includes: sample pupil diameter, sample fixation point coordinates, sample fixation time, and sample eye saccade amplitude, and each sample is added with a label of whether to focus on an icon or a name;

[0023] After the sample EEG signal is subjected to wavelet transform denoising, the average power A of the delta wave at each set time interval is obtained by power spectrum analysis. δ 、θ wave average power A θ , α wave average power A α , β wave average power Aβ , γ wave average power A γ , and calculate (A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β As the sample feature value corresponding to each sample;

[0024] A recurrent neural network model is constructed, with the current sample pupil diameter, the sample gaze point coordinates, the sample gaze time, the sample saccade amplitude, the sample feature value and the hidden state at the previous moment of each sample as input, and the result of whether to pay attention to the icon or name and the hidden layer state at the current moment as output, and the training sample set is used to train and update the parameters of the recurrent neural network model.

[0025] In some embodiments, calculating the correlation coefficient between each reference signal and the EEG data, and obtaining the target reference signal having the largest correlation coefficient with the EEG data, comprises:

[0026] Get the EEG signal X = [x1, x2, ..., x n ], and the reference signal corresponding to the stimulation frequency of each icon or name information, the expression Y i for:

[0027]

[0028] Among them, i represents the i-th icon or name information, f i represents the stimulation frequency of the i-th icon or name information, k represents the harmonic number of the reference signal, N s Indicates the number of sampling points;

[0029] Calculate the correlation coefficient between the EEG signal and the i-th reference signal, and the calculation formula is:

[0030]

[0031] Where x is a linear combination of X, x = X T W x , W x Weight matrix; y is the weight matrix of Y i The linear combination of i T W yi , W yi is the weight matrix; E(·) means to find the expected value;

[0032] The stimulation frequency with the largest output correlation coefficient is the EEG SSVEP frequency, which is expressed as:

[0033]

[0034] Among them, f s represents the EEG SSVEP frequency;

[0035] A reference signal whose stimulation frequency is consistent with the EEG SSVEP frequency is used as a target reference signal.

[0036] In some embodiments, displaying specific parameter information of the target icon or the target name on the cockpit HUD includes:

[0037] Displaying specific parameter information of the target icon or the target name separately in the setting area;

[0038] And / or, specific parameter information of the target icon or the target name is highlighted.

[0039] In some embodiments, after using the pre-trained neural network model to determine whether the operator has an icon or name of interest, the method further includes:

[0040] If it is determined that the operator is not paying attention to the icon or name, the cockpit HUD display content remains unchanged.

[0041] On the other hand, the present invention also provides a cockpit HUD display control system based on human intelligence, characterized in that it includes: an eye movement data acquisition device; an electroencephalogram data acquisition device; a display device; a processor module, the processor module is connected to the eye movement data acquisition device, the electroencephalogram data acquisition device and the display device, and executes the above-mentioned cockpit HUD display control method based on human intelligence.

[0042] On the other hand, the present invention also provides a cockpit HUD display control system based on human factors intelligence, including a processor and a memory, characterized in that computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the above method.

[0043] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program implements the steps of the above method when executed by a processor.

[0044] The beneficial effects of the present invention are at least:

[0045] The cockpit HUD display control method, system and device of the present invention only display the icons or name information of the flight parameter indicators on the flight cockpit HUD in the original state, and the icons or names in each interface interest area are displayed according to different flashing frequencies. By collecting the pilot's EEG signals and eye movement signals, multivariate eigenvalues ​​are constructed, and the pre-trained support vector machine is used to determine whether the pilot is paying attention to the target based on the eigenvalues. On the basis that the pilot pays attention to the target, the target interface interest area of ​​interest is first determined according to the coordinates of the gaze point, and then the corresponding reference signal is calculated according to the stimulation frequency of each icon or name information in the target interface interest area, and the target reference signal with the highest correlation with the EEG signal is identified, and the target icon or target name information corresponding to the target reference signal is obtained, and its parameters are displayed. It can actively identify and display the information that the pilot is concerned about, reduce the pilot's observation pressure, and improve the pilot's information acquisition efficiency.

[0046] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0047] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0049] Figure 1 The figure is a flow chart of a cockpit HUD display control method based on human intelligence according to an embodiment of the present invention.

[0050] Figure 2 The present invention is a schematic diagram of a display interface of a flight cockpit HUD in a cockpit HUD display control method based on human intelligence according to an embodiment of the present invention.

[0051] Figure 3 This is a structural diagram of a recurrent neural network constructed by a cockpit HUD display control method based on human intelligence according to an embodiment of the present invention.

[0052] Figure 4 The schematic diagram of the support vector machine constructed by the cockpit HUD display control method based on human intelligence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0054] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0055] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0056] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0057] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0058] The amount of information that pilots need to receive and process when performing flight missions is very large. However, the HUD interface in the prior art is limited in size and cannot present all the information. The HUD interface does not have a pilot's attention recognition module, and cannot adjust the interface according to the pilot's attention intention. Different pilots have specific operating habits and judgment criteria, and have different requirements for interface information. In the absence of personalized display, the large amount of information displayed on the flight cockpit HUD will cause pressure on pilots to obtain information, and in intense flight missions, it will cause pilots to be highly uncomfortable, fatigued and bored. The present invention provides a cockpit HUD display control method, system and device, which optimize the HUD human-machine interface to reduce information redundancy, reduce pilot workload, ensure flight safety, and reduce human errors.

[0059] Specifically, the present invention provides a cockpit HUD display control method based on human intelligence, such as Figure 1 As shown, the method includes the following steps S101 to S107:

[0060] Step S101: Displaying icons or names of multiple parameter indicators on the cockpit HUD, wherein the cockpit HUD presets multiple interface interest areas, and the icons or names in each interface interest area are displayed at different flashing frequencies.

[0061] Step S102: Obtaining the EEG data and eye movement data of the operator in the cockpit at set time intervals.

[0062] Step S103: using the EEG data and eye movement data as input, using a pre-trained neural network model to determine whether the operator has an icon or name of interest.

[0063] Step S104: If it is determined that the operator has an icon or name of interest, the coordinates of the gaze point in the current time interval are obtained according to the eye movement data, and the target interface interest area of ​​interest of the operator is determined according to the coordinate range of each interface interest area.

[0064] Step S105: Acquire the stimulation frequency of each icon or name in the target interface interest area, and acquire the reference signal corresponding to each stimulation frequency.

[0065] Step S106: Calculate the correlation coefficient between each reference signal and the EEG data, obtain the target reference signal with the largest correlation coefficient with the EEG data, and use the icon or name corresponding to the target reference signal as the target icon or target name.

[0066] Step S107: Display the specific parameter information of the target icon or target name on the cockpit HUD.

[0067] In step S101, only the icon or name information of the specified flight parameter indicator is displayed on the flight cockpit HUD, without displaying the specific parameter value. The flight cockpit HUD presets multiple interface interest areas, and in each interface interest area, the flight parameter indicator uses a different flashing frequency to distinguish each other. Between different interface interest areas, the flight parameter indicator can use the same flashing frequency.

[0068] Exemplarily, an information management system for a flight cockpit HUD is constructed.

[0069] A coordinate system is established for the HUD interface, with the upper left corner of the HUD as the origin (0,0) and the lower right corner as the point (1,1). The HUD interface is divided into different regions of interest (ROIs). For example, the HUD interface can be uniformly divided into 8 regions based on the principle of design specifications and function priority, such as Figure 2 As shown, the upper left corner ① of the interface is the latitude area, which displays the set altitude, air pressure and wind speed and other parameter information; the upper right corner ② is the longitude area, which mainly displays the longitude position parameters, GPS, Mach number, overload and other parameter information; the upper top of the interface ⑦ is the heading area, and the pilot determines the current aircraft heading information based on the information in this area; the horizontal left direction ⑤ is the speed area; the middle ⑧ is the target aiming area, which can display the horizontal pitch angle information, and the target range box will be displayed in combat mode; the horizontal right direction ⑥ is the height and altitude area; the lower left ④ is the mission status area, the lower right ③ is the target parameter area, and the lower bottom of the interface ⑨ is the turn and sideslip indicator display area.

[0070] Each type of icon or text in an interest area of ​​a certain interface must be presented with a different flashing frequency, such as Figure 2 In the middle, the longitude icon in area ② is presented with a flashing frequency of 13.09Hz, and the GPS icon is presented with a flashing frequency of 14.04Hz; different interface interest areas can use a set of flashing frequency combinations of 13.09Hz, 14.04Hz, 16Hz, 18Hz, 20.57Hz, and 24Hz. However, it must be satisfied that the flashing frequencies of different types of information in the same interface interest area cannot be repeated. The flashing frequency is a multiple of the refresh frequency of the flight cockpit HUD display. Specifically, the frequency used for flashing targets is an integer divisor of the display refresh rate, that is, 144 / 11=13.0909, 144 / 10=14.40, 144 / 9=16.00, 144 / 8=18.00, 144 / 7=20.5714 and 144 / 6=24.00Hz.

[0071] Step S102 is the collection of EEG data and eye movement data. Exemplarily, a data collection device is constructed, and a 16-channel non-invasive EEG collection device is used to collect the EEG signal through 8 electrodes respectively set in the parietal and occipital regions on the left and right sides, the ground electrode FPz and the reference electrode on the right ear to record SSVEP. All data are sampled at a frequency of 1200 Hz, and all electrodes are positioned according to the international 10-20 system. The impedance of all electrodes is reduced to less than 1kohm.

[0072] Eye movement data can be collected using an eye tracker. A camera can be used to collect eye movement information of the pilot when viewing the HUD, with a sampling rate of 250 Hz. In some embodiments, in the eye movement data, eye movement points with an eye movement speed threshold greater than 30° / s are marked as scan points, and eye movement points with an eye movement speed threshold less than or equal to 30° / s are marked as fixation points.

[0073] The collection of EEG signals and eye movement data can be divided into two stages. The first stage is the collection of initial information, which is used to pre-train the neural network used to identify whether the pilot is paying attention to a certain target. In this stage, the pilot planned to complete two different HUD interaction tasks: (a) watching each area of ​​interest without a target for 2 minutes; (b) watching each area of ​​interest for 2 minutes with the goal of searching for a certain type of information in the area of ​​interest. The pilot's eye movement data under the two tasks was collected. The second stage is the actual execution of the flight mission, during which the pilot's EEG signals and eye movement data are collected and the interaction strategy of the flight cockpit HUD is implemented.

[0074] In some embodiments, after acquiring the EEG data and eye movement data of the operator in the cockpit at a set time interval, it also includes: pre-processing the eye movement data for noise reduction; applying a 50Hz notch filter to the EEG data, downsampling it to 300Hz, and then performing a bandpass filter to retain 12-110Hz.

[0075] In step S103, the eye movement data may include: pupil diameter, fixation point coordinates, fixation time and saccade amplitude;

[0076] The method may also include: analyzing the EEG data to obtain the average power A of the delta wave at each set time interval δ 、θ wave average power A θ , α wave average power A α , β wave average power A β , γ wave average power A γ , and calculate (A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β as eigenvalues;

[0077] The pupil diameter, fixation point coordinates, fixation time, eye saccade amplitude, A corresponding to the current time interval δ , A θ , A α , A β , A γ ,(A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β As input to the neural network model.

[0078] Specifically, in step S103, feature extraction is performed on the collected EEG signals and eye movement data, the eye movement data is segmented according to a time interval of 1 second, and the x-axis coordinate of the eye movement gaze point (Fx), the y-axis coordinate of the gaze point (Fy), the pupil diameter (PD), the fixation time (FT) and the saccade amplitude (SA) are extracted.

[0079] The collected EEG signals were denoised by wavelet transform, and then the average power of delta wave (frequency range: 0.5-4 Hz), theta wave (frequency range: 4-8 Hz), alpha wave (frequency range: 8-12 Hz), beta wave (frequency range: 12-30 Hz), and gamma wave (frequency range: 30-100 Hz) in each frequency band was obtained through power spectrum analysis.

[0080] For example, the average power of α waves A α The calculation formula is:

[0081]

[0082] Where (a, b) is the upper and lower limits of the α wave frequency band, and P(f) is the power spectrum density of the α wave. The average power of the δ wave A δ , the average power of theta wave A θ , the average power of β wave A β and the average power of the gamma wave A γ The calculation method can refer to the α wave.

[0083] Furthermore, step S103 also introduces (A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β As a feature value, to improve the detection accuracy.

[0084] In this embodiment, a classification task is performed by training a neural network to identify whether the pilot is paying attention to a certain target according to characteristic parameters. In this embodiment, the aforementioned classification task can be performed by using a pre-trained recurrent neural network model (GRU) or building a support vector machine.

[0085] In some embodiments, the neural network model is a recurrent neural network model, and the training steps of the recurrent neural network model include steps S201 to S203:

[0086] Step S201: Obtain a training sample set, which includes multiple samples. Each sample includes multi-channel sample EEG data and sample eye movement data of an operator within a specified time period. The sample EEG data includes sample EEG signals, and the sample eye movement data includes: sample pupil diameter, sample gaze point coordinates, sample gaze time, and sample eye movement amplitude. Each sample is labeled with whether to focus on an icon or name.

[0087] Step S202: After performing wavelet transform to denoise the sample EEG signal, the average power A of the delta wave at each set time interval is obtained by power spectrum analysis.δ 、θ wave average power A θ , α wave average power A α , β wave average power A β , γ wave average power A γ , and calculate (A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β As the sample feature value corresponding to each sample.

[0088] Step S203: Construct a recurrent neural network model, using the current sample pupil diameter, sample fixation point coordinates, sample fixation time, sample saccade amplitude, sample feature value and the hidden state of the previous moment for each sample as input, and using the result of whether to pay attention to the icon or name and the hidden layer state at the current moment as output, and use the training sample set to train and update the parameters of the recurrent neural network model.

[0089] Specifically, the input of the recurrent neural network model consists of two parts: an input x at time t t , and the other is the hidden layer state h at time t-1 t-1 , this hidden layer state contains the relevant information of the previous node; the output also includes two parts, one is the judgment result y at the current time t t , and the other is the hidden state at the current time t.

[0090] like Figure 3 As shown, the recurrent neural network constructs the states of five gates, including forget gate, learning gate, update gate, transmission gate and prediction gate.

[0091] Among them, the calculation formula of the forget gate is:

[0092] r t =δ(x t W xr +h t-1 W hr +b r )

[0093] In the above formula, x t is the input vector at the current moment, h t-1 is the hidden state at the previous moment, W xr , W hr is the learning weight, b r is the forget gate bias, δ is the sigmoid activation function, which maps each component to a value between 0 and 1; the forget gate can be used to adjust the hidden state h t-1Filter the data in t is the forget vector;

[0094] The calculation formula of the learning gate is:

[0095] t t =tanh(x t W xt +(r t *h t-1 )W rht +b t )

[0096] In the above formula, W xt , W rhr is the learning weight, b t is the learning gate bias, tanh is the activation function, t t is the learned vector; this is an input vector x at the current moment t and the previous hidden state h t-1 (existing knowledge) The process of learning new knowledge;

[0097] The calculation formula of the update gate is:

[0098] z t =δ(x t W xz +h t-1 W hz +b z )

[0099] In the above formula, W xz , W hz is the learning weight, b z To update the gate bias, since the network parameters are fixed, the knowledge that can be learned is limited. In this case, the update gate can determine the hidden state h at the previous moment t-1 and the learned vector t t The respective proportions in the information fusion process; t To update the scale vector;

[0100] The calculation formula for the portal is:

[0101] h t =(1-z t )*h t-1 +z t *t t

[0102] In the above formula, h t is the hidden state of the current time step; this hidden state will be used as the previous time step in the calculation and prediction of the next time step;

[0103] The calculation formula of the prediction gate is:

[0104] y t =δ(Wh t )

[0105] In the above formula, W is the prediction weight vector; since it is a binary classification task, the sigmoid activation function is still used to predict the final result. When the activation value is greater than 0.5, the final prediction result y t is "attention state", when the activation value is less than 0.5, the final prediction result y t If the target is being paid attention to, the output is 1; if the target is not being paid attention to, the output is 0.

[0106] The data used for training the support vector machine may be constructed based on the initial information collected in the first stage in step S102.

[0107] Specifically, the pre-training step of the support vector machine includes steps S301 to S304:

[0108] Step S301: Acquire multiple samples, each sample including a multi-channel electroencephalogram signal and eye movement data collected from at least one sample pilot within a set time interval, the electroencephalogram signal at least including an EEG signal; the eye movement data at least including: pupil diameter, fixation time and saccade amplitude.

[0109] Step S302: After performing wavelet transform to denoise the EEG signal of each sample, the average power A of the delta wave corresponding to each sample is obtained by power spectrum analysis. δ 、θ wave average power A θ , α wave average power A α , β wave average power A β , γ wave average power A γ , and calculate (A α +A β ) / A θ ,A α / A β ,(A α +A θ ) / A β as a feature value.

[0110] Step S303: The pupil diameter, fixation time, saccade amplitude, and A corresponding to each sample are used. δ , A θ , A α , A β , A γ ,(A α +A β ) / A θ , A α / A β and (Aα +A θ ) / A β Construct a training sample set and add a label to each sample to mark whether it focuses on the target. If it focuses on the target, add label 1; if it does not focus on the target, add label 0.

[0111] Step S304: construct a support vector machine using the training sample set to identify whether the sample pilot is paying attention to the determined target.

[0112] Exemplarily, in steps S301 to S304, the training sample concentrates on the protected feature vector X(t) and the label quantity Label(t), where X(t) = {x1(t), x2(t), ..., x i (t)}, Label(t)={y(t)}.

[0113] In the formula, x1(t), x2(t), ..., x i (t) respectively represent the information of each feature quantity corresponding to the time window T, which can be pupil diameter, fixation time, saccade amplitude, A δ , A θ , A α , A β , A γ ,(A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β y(t) is the intention label, which is represented by a number. For example, 0 means that the pilot is not paying attention to the target in the current period, and 1 means that the pilot is paying attention to the target in the current period. The time window used in the current period is 1 second.

[0114] In the process of training the model, the feature vectors of eye movement data and EEG signals are imported into the support vector machine SVM for training. Given L sample data (x1, y1), (x2, y2), ..., (x l ,y l )∈R n ×R,x i is a set of feature vectors consisting of input sample feature parameters; i is the sample category label set. The support vector machine SVM model is used to train based on the training sample set and establish the hyperplane W T X+b=0, where X is the input vector, W is the hyperplane normal vector, b is the bias term, and T represents the inversion of the vector. Figure 4 shown.

[0115] The decision function of SVM is:

[0116]

[0117] Among them, a * i represents the Lagrange multiplier, b represents the bias term of the hyperplane, K represents an arbitrary kernel function, x i Represents the information of the corresponding feature quantity in the i-th time window; y i Represents the intent label at the i-th time window.

[0118] The optimization equation is:

[0119]

[0120] In the formula, a * i represents the Lagrange multiplier, K represents an arbitrary kernel function, x i Represents the information of the corresponding feature quantity in the i-th time window; y i represents the intention label in the i-th time window, C is the selected value penalty parameter; K(x i ,x j ) is an arbitrary kernel function.

[0121] The support vector machine obtained through training can realize intention recognition, import the features of eye movement data and electroencephalogram signals, and when the output y=0, it is judged that the pilot has no definite target; when the output y=1, it is judged that the pilot has a definite target.

[0122] In step S104, when the neural network model of step S103 outputs the judgment result y=1 of the current period, it is judged that the pilot is paying attention to a certain target, and further judged which interface interest area the pilot is paying attention to. The judgment of the interface interest area of ​​interest is mainly based on the coordinates of the gaze point as a reference. In some embodiments, in the eye movement data, the eye movement points with an eye movement speed threshold greater than 30° / s are marked as scanning points, and the eye movement points with an eye movement speed threshold less than or equal to 30° / s are marked as fixation points.

[0123] Specifically, let the coordinate range of the i-th interface region of interest ROI be i ((x min ,X max ), (y min ,y max )). For the gaze point coordinates (Fx, Fy), if X min <Fx<X max , and y min <Fy<y max , the position of the gaze point can be determined to be within the i-th interface interest area.

[0124] In steps S105 and S106, after the interface interest area that the pilot is looking at is determined, the icon or text information corresponding to the specific flight parameter indicator that the pilot is concerned about is further determined. Specifically, the judgment is made by determining the EEG SSVEP frequency. Early state visual evoked potentials (SSVEPs) have the advantages of good robustness, large instruction volume, high classification accuracy, and high information transmission rate, and are widely used in the development of brain-computer interfaces. In a single interface interest area, the icons or text information corresponding to each flight parameter indicator are displayed according to different flashing frequencies, and the stimulation generated for the pilot is not the same. By finding the EEG SSVEP frequency related to the pilot's EEG signal, the frequency of the icon or text information that the pilot is concerned about can be determined, and thus the specific icon or text information that he is concerned about can be determined.

[0125] In some embodiments, calculating the correlation coefficient between each reference signal and the EEG data, and obtaining the target reference signal with the largest correlation coefficient with the EEG data, includes steps S401 to S404:

[0126] Step S401: Obtain the EEG signal X = [x1, x2, ..., x n ], and the reference signal corresponding to the stimulation frequency of each icon or name information, the expression Y i for:

[0127]

[0128] Among them, i represents the i-th icon or name information, f i represents the stimulation frequency of the i-th icon or name information, k represents the harmonic number of the reference signal, N s Indicates the number of sampling points.

[0129] Step S402: Calculate the correlation coefficient between the EEG signal and the i-th reference signal, using the following formula:

[0130]

[0131] Where x is a linear combination of X, x = X T W x , W x Weight matrix; y is the weight matrix of Y i The linear combination of i T W yi , W yi is the weight matrix; E(·) means to find the expected value.

[0132] Step S403: The stimulation frequency with the largest output correlation coefficient is the EEG SSVEP frequency, which is expressed as:

[0133]

[0134] Among them, f s Represents the EEG SSVEP frequency.

[0135] Step S404: taking a reference signal whose stimulation frequency is consistent with the EEG SSVEP frequency as a target reference signal.

[0136] In step S107, since the flight cockpit HUD does not display specific parameters in icons or text information in its original state, after identifying the flight parameter indicators that the pilot intends to pay attention to, it further displays them. In this way, the flight cockpit HUD always only displays the parameters that the pilot wants to see, simplifies the display interface, realizes information filtering, and reduces the recognition pressure of the pilot.

[0137] In some embodiments, specific parameter information of the target icon or target name is displayed on the cockpit HUD, including:

[0138] In the setting area, specific parameter information of the target icon or target name is displayed separately;

[0139] And / or, specific parameter information of the target icon or target name is highlighted.

[0140] In some embodiments, after step S104, that is, after using a pre-trained neural network model to determine whether the operator has paid attention to an icon or name, the method further includes: if it is determined that the operator has not paid attention to an icon or name, keeping the cockpit HUD display content unchanged.

[0141] On the other hand, the present invention also provides a cockpit HUD display control system based on human intelligence, characterized in that it includes: an eye movement data acquisition device; an electroencephalogram data acquisition device; a display device; a processor module, the processor module is connected to the eye movement data acquisition device, the electroencephalogram data acquisition device and the display device, and executes the above-mentioned cockpit HUD display control method based on human intelligence.

[0142] An embodiment of the present invention further provides a computer device, which may include a processor and a memory, wherein the processor and the memory may be connected via a bus or in other ways.

[0143] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0144] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the key shielding method of the vehicle-mounted display device in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory.

[0145] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The one or more modules are stored in the memory, and when executed by the processor, the method described in this embodiment is performed.

[0147] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0148] In summary, the cockpit HUD display control method, system and device of the present invention only displays the icons or name information of the flight parameter indicators on the flight cockpit HUD in the original state, and the icons or names in each interface interest area are displayed according to different flashing frequencies. By collecting the pilot's EEG signals and eye movement signals, a multivariate eigenvalue is constructed, and the pre-trained support vector machine is used to determine whether the pilot is paying attention to the target based on the eigenvalue. On the basis that the pilot pays attention to the target, the target interface interest area of ​​interest is first determined according to the coordinates of the gaze point, and then the corresponding reference signal is calculated according to the stimulation frequency of each icon or name information in the target interface interest area, and the target reference signal with the highest correlation with the EEG signal is identified, and the target icon or target name information corresponding to the target reference signal is obtained, and its parameters are displayed. It can actively identify and display the information that the pilot is concerned about, reduce the pilot's observation pressure, and improve the pilot's information acquisition efficiency.

[0149] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0150] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0151] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cockpit HUD display control method based on human intelligence, characterized in that: The method comprises the following steps: Displaying icons or names of multiple parameter indicators on the cockpit HUD, wherein the cockpit HUD presets multiple interface interest areas, and the icons or names in each interface interest area are displayed according to different flashing frequencies; Acquiring EEG data and eye movement data of the operator in the cockpit at set time intervals; Using the EEG data and the eye movement data as input, a pre-trained neural network model is used to determine whether the operator has an icon or name of interest; If it is determined that the operator has an icon or name of interest, the coordinates of the gaze point in the current time interval are obtained according to the eye movement data, and the target interface interest area of ​​interest of the operator is determined according to the coordinate range of each interface interest area; Obtaining the stimulation frequency of each icon or name in the target interface interest area, and obtaining a reference signal corresponding to each stimulation frequency; Calculate the correlation coefficient between each reference signal and the EEG data, obtain the target reference signal with the largest correlation coefficient with the EEG data, and use the icon or name corresponding to the target reference signal as the target icon or target name; Displaying specific parameter information of the target icon or the target name on the cockpit HUD; Wherein, the eye movement data includes: pupil diameter, fixation point coordinates, fixation time and eye saccade amplitude; The method further comprises: analyzing the EEG data to obtain the average power A of the delta wave at each set time interval δ 、θ wave average power A θ , α wave average power A α , β wave average power A β , γ wave average power A γ , and calculate (A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β As feature values; the pupil diameter, the fixation point coordinates, the fixation time, the eye saccade amplitude, the A corresponding to the current time interval δ , said A θ , said A α , said A β , said A γ ,(A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β As input to the neural network model; The neural network model is a recurrent neural network model, and the training steps of the recurrent neural network model include: Acquire a training sample set, wherein the training sample set includes multiple samples, each sample includes multi-channel sample EEG data and sample eye movement data of an operator within a specified time period, the sample EEG data includes a sample EEG signal, and the sample eye movement data includes: sample pupil diameter, sample fixation point coordinates, sample fixation time, and sample eye saccade amplitude, and each sample is added with a label of whether to focus on an icon or a name; After the sample EEG signal is subjected to wavelet transform denoising, the average power A of the delta wave at each set time interval is obtained by power spectrum analysis. δ 、θ wave average power A θ , α wave average power A α , β wave average power A β , γ wave average power A γ , and calculate (A α +A β ) / A θ , A α / A β and (A α +A θ ) / A β As the sample feature value corresponding to each sample; A recurrent neural network model is constructed, with the current sample pupil diameter, the sample gaze point coordinates, the sample gaze time, the sample saccade amplitude, the sample feature value and the hidden state at the previous moment of each sample as input, and the result of whether to pay attention to the icon or name and the hidden layer state at the current moment as output, and the training sample set is used to train and update the parameters of the recurrent neural network model.

2. The cockpit HUD display control method based on human intelligence according to claim 1 is characterized in that: The icons or names in each interface interest area are displayed at different flashing frequencies, and the flashing frequency is a multiple of the cockpit HUD display refresh frequency.

3. The cockpit HUD display control method based on human intelligence according to claim 1 is characterized in that: In the eye movement data, the eye movement points with an eye movement speed threshold greater than 30° / s are marked as saccade points, and the eye movement points with an eye movement speed threshold less than or equal to 30° / s are marked as fixation points.

4. The cockpit HUD display control method based on human intelligence according to claim 1 is characterized in that: After obtaining the EEG data and eye movement data of the operator in the cockpit at a set time interval, the method further includes: Performing noise reduction preprocessing on the eye movement data; The EEG data was subjected to 50 Hz notch filtering, down-sampled to 300 Hz, and then band-pass filtered to retain 12-110 Hz.

5. The cockpit HUD display control method based on human intelligence according to claim 1 is characterized in that: Calculating the correlation coefficient between each reference signal and the EEG data, and obtaining the target reference signal having the largest correlation coefficient with the EEG data, including: Get the EEG signal X = [x1, x2, ..., x n ], and the reference signal corresponding to the stimulation frequency of each icon or name information, the expression Y i for: Among them, i represents the i-th icon or name information, f i represents the stimulation frequency of the i-th icon or name information, k represents the harmonic number of the reference signal, N s Indicates the number of sampling points; Calculate the correlation coefficient between the EEG signal and the i-th reference signal, and the calculation formula is: Where x is a linear combination of X, x = X T W x , W x Weight matrix; y is the weight matrix of Y i The linear combination of i T W yi , W yi is the weight matrix; E(·) means to find the expected value; The stimulation frequency with the largest output correlation coefficient is the EEG SSVEP frequency, which is expressed as: Among them, f s represents the EEG SSVEP frequency; A reference signal whose stimulation frequency is consistent with the EEG SSVEP frequency is used as a target reference signal.

6. The cockpit HUD display control method based on human intelligence according to claim 1 is characterized in that: Displaying specific parameter information of the target icon or the target name on the cockpit HUD includes: Displaying specific parameter information of the target icon or the target name separately in the setting area; And / or, specific parameter information of the target icon or the target name is highlighted.

7. The cockpit HUD display control method based on human intelligence according to claim 1 is characterized in that: After using the pre-trained neural network model to determine whether the operator has an icon or name of interest, the method further includes: If it is determined that the operator is not paying attention to the icon or name, the cockpit HUD display content remains unchanged.

8. A cockpit HUD display control system based on human intelligence, characterized in that: include: Eye movement data collection equipment; EEG data acquisition equipment; Display devices; A processor module, wherein the processor module is connected to the eye movement data acquisition device, the electroencephalogram data acquisition device and the display device, and executes the cockpit HUD display control method based on human intelligence as described in any one of claims 1 to 7.

9. A cockpit HUD display control system based on human intelligence, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images