A Deep Learning-Based Attention Detection Method

Abstract

An attention detection method based on deep learning, comprising the following steps: optimizing parameters a and r; inputting a face image I(t) into a deep neural network to output a line-of-sight direction {pitch(t), yaw(t)}; calculating an attention A(t) = |pitch(t)<supgt;a< / supgt> + yaw(t)<supgt;a< / supgt>|<supgt;r< / supgt>; and filtering and smoothing A(t) to obtain an attention signal L(t).

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and specifically relates to a method for obtaining an attention signal by using the line-of-sight direction obtained from a deep learning network. Background Art

[0002] Attention is a mental activity of human beings, which is the direction and concentration on a certain object; attention is the ability to direct and concentrate mental activities on a certain object. Attention is the gateway to regulating information into the center of consciousness. Precise perception and effective control of attention are effective methods for individuals to improve learning and work efficiency, relieve stress, improve mood, and detect abnormalities.

[0003] Currently, the methods for attention detection include questionnaire method, electroencephalogram signal-based method, and virtual reality-based method. These methods are either unable to be quantified, or have high hardware requirements, or complex hardware requirements, and there is no non-invasive, objective, and real-time inspection method that is not affected by head posture. Summary of the Invention

[0004] The purpose of the present invention is to provide a non-invasive, objective, and real-time inspection method that is not affected by head posture.

[0005] To achieve the above purpose, an attention detection method based on deep learning according to the present invention includes the following steps:

[0006] 1. An attention detection method based on deep learning, characterized by including the following steps:

[0007] Step 1, optimize parameters a and r by using the following 6 steps:

[0008] ① Collect face image I0(t) under standard conditions, and initialize 5 parameters k = 1, a0 = 0, r0 = 0, △a = 0.1,

[0009] △r = 0.02;

[0010] ① Input face image I0(t) into a deep neural network, and the deep neural network is DenseNet-88, to obtain the line-of-sight direction {pitch(t), yaw(t)};

[0011] ③ Calculate parameters a and r by using the following formula:

[0012] a k = a k-1 +Δa, r k = r k-1 +Δr;

[0013] ④ Calculate attention by using the following formula:

[0014]

[0015] ⑤ Calculate the signal-to-noise ratio using the following formula:

[0016] SNR = 10 log 10 [(∑A0(t) 2 ) / |∑A(t) 2 - ΣA(t) 2 |],

[0017] where the reference signal is:

[0018] A0(t) = 0.5 sin(0.5πt) + 0.5;

[0019] ⑥ Determine whether the SNR is greater than 15 dB. If it is, output the optimized parameters a and r. Otherwise, k = k + 1 and go to ③.

[0020] Step 2: Input the face image I(t) into the DenseNet-88 deep learning network, and the network outputs the line-of-sight directions {pitch(t), yaw(t)};

[0021] Step 3: Calculate the attention according to the following formula:

[0022] A(t) = |pitch(t) a + yaw(t) a | r ;

[0023] Step 4: Filter A(t) to obtain B(t), and smooth B(t) to obtain the attention signal L(t).

[0024] 2. A deep learning-based attention detection method according to claim 1, wherein the formula for calculating the attention in step 3 is:

[0025] A(t) = |pitch(t) a + yaw(t) a | r

[0026] where a and r are affected by factors such as the focal length, working distance, aberration, and spherical aberration of the imaging system, and the measurement accuracy can be improved after optimization.

[0027] 3. A deep learning-based attention detection method according to claim 1, wherein in step 4, filtering and smoothing A(t) to obtain the attention signal L(t) specifically adopts the following steps:

[0028] Step 1: Use Kalman filtering on A(t) to obtain B(t);

[0029] Step 2: Smooth B(t) through a Savitzky-Golay filter to obtain the attention signal L(t). Description of the Drawings

[0030] Figure 1 is the flowchart of this patent;

[0031] Figure 2 is the flowchart for optimizing parameters a and r;

[0032] Figure 3 is the device diagram for attention detection;

[0033] Figure 4 is the structural diagram of the deep learning network;

[0034] Figure 5 is the original attention signal A(t);

[0035] Figure 6 is the signal B(t) after A(t) is filtered by the Kalman filter;

[0036] Figure 7 is the signal after B(t) is smoothed. Detailed Implementation Manner

[0037] The present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0038] A specific implementation of attention detection based on deep learning is as Figure 3 shown, including a computer PC and a camera. The hardware environment of the PC is Intel(R) Core(TM) i9-9900K, NVIDIA GTX1080Ti, DDR4 16GB, and a 500GB hard disk; the software environment is the Ubuntu16.04 operating system, Python3.7.4, and the Pytorch1.4.0 deep learning framework. The frame rate of the camera is 20fps, and the resolution is 1280*720.

[0039] The camera is placed at the upper edge of the computer monitor. The participant is about 60 cm away from the screen and gazes at the screen in a natural sitting posture.

[0040] The attention detection process based on deep learning is as Figure 1 shown, including the following steps:

[0041] Step 1: Optimize parameters a and r according to the process as Figure 2 follows: At the initial moment, the fixation point is in the center of the screen and moves horizontally at a constant speed, first to the right and then to the left, ensuring a cycle of 4 seconds; typical values of the optimized parameters are a = 2 and r = 0.8;

[0042] Step 2: Input the face image I(t) into asFigure 4 The DenseNet-88 deep learning network shown to obtain the line-of-sight directions {pitch(t), yaw(t)};

[0043] Step 3, calculate the attention according to the following formula:

[0044] A(t) = |pitch(t) a + yaw(t) a | r

[0045] The obtained signal is Figure 5 the original attention signal shown, and this signal inevitably contains noise and distortion;

[0046] Step 4, perform Kalman filtering on the A(t) signal to obtain B(t) as Figure 6 shown;

[0047] Step 5, perform Savitzky-Golay smoothing on B(t) to obtain the attention signal L(t) as Figure 7 shown.

Claims

1. An attention detection method based on deep learning, characterized in that, It includes the following steps: Step 1: Optimize parameters a and r using the following 6 steps: ① Collect a face image I0(t) under standard conditions, and initialize 5 parameters: k = 1, a0 = 0, r0 = 0, △a = 0.1, △r = 0.02; ② Input the face image I0(t) into a deep neural network, which is DenseNet-88, to obtain the line-of-sight directions {pitch(t), yaw(t)}; ③ Calculate the a and r parameters using the following formula: a k = a k-1 + Δa,r k = r k-1 + Δr; ④ Calculate the attention using the following formula: ⑤ Calculate the signal-to-noise ratio using the following formula: Among them, the reference signal is: A0(t) = 0.5sin(0.5πt) + 0.5; ⑥ Determine whether the SNR is greater than 15 dB. If so, output the optimized parameters a and r. Otherwise, k = k + 1 and go to ③. Step 2: Input the face image I(t) into the DenseNet-88 deep learning network, and this network outputs the line-of-sight directions {pitch(t), yaw(t)}; Step 3: Calculate the attention according to the following formula: A(t) = |pitch(t) a + yaw(t) a | r ; Step 4: Filter A(t) to obtain B(t), and smooth B(t) to obtain the attention signal L(t).

2. The attention detection method based on deep learning according to claim 1, wherein The formula for calculating the attention in Step 3 is: A(t) = |pitch(t) a + yaw(t) a | r Among them, a and r are affected by factors such as the focal length, working distance, aberration, and spherical aberration of the imaging system. After optimization, the measurement accuracy can be improved.

3. The attention detection method based on deep learning according to claim 1, characterized in that, In Step 4, the attention signal L(t) is obtained by filtering and smoothing A(t). Specifically, the following steps are adopted: Step 1: Use a Kalman filter on A(t) to obtain B(t); Step 2: Smooth B(t) using a Savitzky-Golay filter to obtain the attention signal L(t).

Images