Micro-eye jump detection method and device, near-eye display equipment and storage medium

Abstract

The invention discloses a micro eye jump detection method and device, near-eye display equipment and a computer readable storage medium. The micro eye jump detection method comprises the following steps: acquiring original data, wherein the original data comprises a time-varying coordinate sequence of an eyeball fixation point; processing the original data based on the filter bank to judge whether micro-eye jump motion exists in the coordinate sequence or not; and controlling the near-to-eye display equipment to execute corresponding operation according to the judgment result. Therefore, noise interference can be effectively avoided, the detection accuracy is improved, the calculation overhead can be reduced, and real-time detection is realized.

Description

Technical Field

[0001] The present application relates to the field of eye movement technology, and in particular to a microsaccade detection method, a detection device, a near-eye display device, and a computer-readable storage medium. Background Art

[0002] Microsaccades are widely connected with behavior, cognition, and neurology. Therefore, how to accurately detect microsaccades is of great research and application value. Microsaccades have a strong correlation with spike pulse neural signals in the visual pathway and are the physiological basis of microsaccades. The rapid movement of microsaccades makes them equivalent to a random perturbation noise superimposed on the human eye's vision, controlling the translation of the image on the retina, thereby offsetting retinal fatigue and enhancing the human perception of objects. In addition, microsaccades are inhibited by concentration, so microsaccades can be used to characterize the tester's visual attention to an object, which is of great application value in the field of new near-eye displays. However, current microsaccade detection methods are easily affected by noise interference and have poor detection results. Summary of the Invention

[0003] The embodiments of the present application provide a micro-saccade detection method, a detection device, a near-eye display device, and a computer-readable storage medium to solve at least one of the above-mentioned technical problems.

[0004] The microsaccade detection method of the embodiment of the present application includes:

[0005] Acquiring raw data, wherein the raw data includes a coordinate sequence of an eye gaze point that changes over time;

[0006] Processing the raw data based on a filter bank to determine whether there is microsaccade movement in the coordinate sequence;

[0007] The near-eye display device is controlled to perform corresponding operations based on the judgment result.

[0008] In some embodiments, obtaining the raw data includes: obtaining the raw data corresponding to the binoculars;

[0009] The processing of the raw data based on the filter bank to determine whether there is microsaccade movement in the coordinate sequence includes:

[0010] Processing the original data corresponding to the binocular vision through the filter group to obtain high-frequency components;

[0011] The high-frequency component is processed to determine whether microsaccades exist in the coordinate sequence.

[0012] In certain embodiments, the coordinate sequence comprises a horizontal coordinate sequence;

[0013] The step of processing the original data corresponding to the binocular eyes by the filter bank to obtain high-frequency components includes:

[0014] The horizontal coordinate sequences corresponding to the binocular eyes are respectively used as items to be transformed;

[0015] Perform discrete wavelet transform on the item to be transformed to obtain high-frequency components.

[0016] In some embodiments, processing the high-frequency component to determine whether microsaccades exist in the coordinate sequence includes:

[0017] Performing filtering and reconstruction processing on the high-frequency component to obtain a reconstructed waveform;

[0018] Post-processing the reconstructed waveform to determine the singular point moment in the coordinate sequence according to micro-saccade characteristics;

[0019] Performing similarity calculation on the singular point moments in the coordinate sequence corresponding to the two eyes to determine the moment when the eye movement occurs;

[0020] Judging whether the microsaccade movement exists in the coordinate sequence according to the time when the eye movement occurs.

[0021] In some embodiments, performing similarity calculation on the singular point moments in the coordinate sequence corresponding to the two eyes to determine the moment when the eye movement occurred includes:

[0022] Determine whether the singular point moment in the coordinate sequence corresponding to the binocular eyes is within a predetermined time window;

[0023] When the singular point moment in the coordinate sequence corresponding to the binocular eyes is within the predetermined time window, the eye movement occurrence moment is determined according to the singular point moment in the coordinate sequence corresponding to the binocular eyes.

[0024] In some embodiments, determining whether the microsaccade movement exists in the coordinate sequence according to the time when the eye movement occurs includes:

[0025] Extracting eye movement segments within a predetermined time period before and after the eye movement occurrence moment from the coordinate sequence;

[0026] Determining whether the eye movement segment conforms to a predetermined movement trajectory;

[0027] When the eye movement segment conforms to the predetermined movement trajectory, it is determined that the microsaccade movement exists in the coordinate sequence.

[0028] In some embodiments, controlling the near-eye display device to perform corresponding operations according to the judgment result includes:

[0029] When the microsaccade movement exists in the coordinate sequence, obtaining a microsaccade segment corresponding to the microsaccade movement;

[0030] Extracting microsaccade feature values ​​from the microsaccade segment;

[0031] The near-eye display device is controlled to perform corresponding operations according to the micro-saccade feature value.

[0032] In some embodiments, controlling the near-eye display device to perform a corresponding operation based on the micro-saccade feature value includes:

[0033] Comparing the microsaccade feature value with a predetermined feature value to determine whether the microsaccade feature value conforms to a feature value distribution of an authorized user, wherein the predetermined feature value is extracted from an eye movement segment entered by the authorized user;

[0034] When the micro-saccade characteristic value meets the authorized user characteristic value distribution, controlling the near-eye display device to perform a screen unlocking operation;

[0035] When the micro-saccade feature value does not conform to the authorized user feature value distribution, the near-eye display device is controlled to keep locking the screen.

[0036] In some embodiments, controlling the near-eye display device to perform corresponding operations according to the judgment result includes:

[0037] When the micro-saccade movement exists in the coordinate sequence, acquiring the occurrence frequency, occurrence time and / or field of view position of the micro-saccade movement in the current screen display content scene of the near-eye display device;

[0038] The near-eye display device is controlled to perform corresponding operations according to the frequency, time of occurrence and / or field of view position of the micro-saccade movement.

[0039] In some embodiments, controlling the near-eye display device to perform corresponding operations based on the frequency, time of occurrence, and / or field of view position of the microsaccade movement includes:

[0040] Determining the degree to which the current screen display content attracts the user's attention based on the frequency, time of occurrence, and / or visual field position of the microsaccade movement;

[0041] When the attraction level increases, controlling the near-eye display device to display screen display content associated with the current screen display content;

[0042] When the attraction level decreases, the near-eye display device is controlled to adjust the current screen display content.

[0043] The microsaccade detection device of the embodiment of the present application includes:

[0044] An acquisition module, configured to acquire raw data, wherein the raw data includes a coordinate sequence of an eye gaze point that changes over time;

[0045] a determination module, configured to process the raw data based on a filter bank to determine whether microsaccades exist in the coordinate sequence;

[0046] The execution module is used to control the near-eye display device to perform corresponding operations according to the judgment result.

[0047] The near-eye display device of an embodiment of the present application includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the micro-saccade detection method of any of the above embodiments is implemented.

[0048] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the program is executed by a processor, the micro-saccade detection method of any of the above embodiments is implemented.

[0049] The microsaccade detection method, detection apparatus, near-eye display device, and computer-readable storage medium of the present application embodiment acquire raw data comprising a coordinate sequence of eye gaze points that changes over time, process the raw data using a filter bank to determine whether microsaccades are present in the coordinate sequence, and control the near-eye display device to perform corresponding operations based on the determination result. This effectively avoids noise interference, improves detection accuracy, reduces computational overhead, and achieves real-time detection.

[0050] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0052] Figure 1 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0053] Figure 2 is a schematic diagram of a module of a near-eye display device according to certain embodiments of the present application;

[0054] Figure 3 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0055] Figure 4is a schematic diagram of a coordinate sequence of the left eye's gaze point changing over time in certain embodiments of the present application;

[0056] Figure 5 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0057] Figure 6 Schematic diagram of the working process of the microsaccade detection method in certain embodiments of the present application;

[0058] Figure 7 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0059] Figure 8 is a schematic diagram of binocular corresponding reconstructed waveforms in certain embodiments of the present application;

[0060] Figure 9 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0061] Figure 10 is a schematic diagram of a singular point moment in a coordinate sequence corresponding to a left eye in certain embodiments of the present application;

[0062] Figure 11 is a schematic diagram of a singular point moment in a coordinate sequence corresponding to a right eye in certain embodiments of the present application;

[0063] Figure 12 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0064] Figure 13 is a schematic diagram of the gaze point movement trajectory of certain embodiments of the present application;

[0065] Figure 14 is a schematic diagram of a quantitative analysis of the speed of the detection results of certain embodiments of the present application;

[0066] Figure 15 2 is a schematic diagram comparing the detection results of the microsaccade detection method according to certain embodiments of the present application with the detection results of related technologies;

[0067] Figure 16 2 is a schematic diagram comparing the detection results of the microsaccade detection method according to certain embodiments of the present application with the detection results of related technologies;

[0068] Figure 17 Schematic diagram of the comparison of singular point moments in a binocular corresponding coordinate sequence in certain embodiments of the present application;

[0069] Figure 18 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0070] Figure 19 This is a flow chart of the microsaccade detection method according to certain embodiments of the present application applied to biometric identification of a near-eye display device;

[0071] Figure 20 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0072] Figure 21 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0073] Figure 22 is a flowchart of applying the microsaccade detection method of certain embodiments of the present application to a recommendation algorithm;

[0074] Figure 23 1 is a flow chart of a microsaccade detection method according to certain embodiments of the present application;

[0075] Figure 24 1 is a schematic diagram of a module of a microsaccade detection device according to certain embodiments of the present application;

[0076] Figure 25 This is a schematic diagram of the connection status between a computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0077] The following further describes the embodiments of the present application in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions. Furthermore, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are intended only to explain the embodiments of the present application and are not to be construed as limiting the present application.

[0078] See also Figure 1 and Figure 2 , the embodiment of the present application provides a micro-saccade detection method, comprising:

[0079] 010: Get the original data, which includes the coordinate sequence of the eye gaze point changing over time;

[0080] 020: Process the raw data based on the filter bank to determine whether there are microsaccades in the coordinate sequence;

[0081] 030: Control the near-eye display device 100 to perform corresponding operations according to the judgment result.

[0082] The microsaccade detection method of the present embodiment acquires raw data, including a coordinate sequence of the eye's gaze point changing over time, processes the raw data using a filter bank to determine whether microsaccades are present in the coordinate sequence, and controls the near-eye display device 100 to perform corresponding operations based on the determination result. This effectively avoids noise interference, improves detection accuracy, reduces computational overhead, and enables real-time detection.

[0083] Microsaccades are a type of fixational eye movement (FEM), with a gaze point moving at a speed of 0.5 to 10° / s. The maximum peak velocity of microsaccades is linearly related to the amplitude of the movement.

[0084] The raw data can be obtained through an eye tracker, which captures the user's eye gaze trajectory in real time and selects the eye gaze trajectory within a certain time period (for example, within 50ms) as the raw data. The raw data includes the coordinate sequence of the eye gaze point changing over time, and the coordinate sequence can be expressed as (x(t), y(t)).

[0085] The filter group is composed of a plurality of filters connected in series, each filter having a different time window, and the output signal of the previous filter is used as the input signal of the next filter. The filter group can be generated by a known wavelet basis function theory. For example, a known wavelet function is used as a discrete wavelet basis function to obtain a discrete wavelet (which will be introduced in detail later) as a filter group. At this time, the filter group has good mathematical properties, such as orthogonality, compactness, smoothness, etc. The filter group can also be generated by machine learning to generate a new wavelet basis function. It can also be separated from the wavelet basis function theory and generated by the current motion state, such as using the coordinate sequence of a single eye movement to generate a filter group of the coordinate sequence of another eye movement, etc., which is not limited here. Based on the processing of the original data by the filter group, it can be determined whether there is micro-saccade movement in the coordinate sequence. According to the judgment result of the micro-saccade movement, the near-eye display device 100 can be controlled to perform corresponding operations.

[0086] In one example, an eye tracker can be installed or integrated into the near-eye display device 100. The eye tracker captures the user's eye gaze trajectory in real time and obtains raw data. The near-eye display device 100 includes a processor 110. The processor 110 can execute a pre-deployed filter bank detection program to process the raw data based on the filter bank, determine whether microsaccades are present in the coordinate sequence, and control the near-eye display device 100 to perform corresponding operations based on the determination result.

[0087] In the related art, a speed threshold method is used to detect microsaccades. By judging whether the speed of the gaze point is higher than the speed threshold, the detection of microsaccades is completed. This method is easily affected by large external low-frequency noise, such as head shaking, and has strict requirements on the detection environment. At the same time, this method also requires adjustment of the hyperparameters of the speed threshold, and the detection process is complicated. In the embodiment of the present application, a filter group is used for microsaccade detection, and the filter group includes multi-stage filtering and downsampling. In this way, noise interference can be effectively avoided, the detection accuracy can be improved, and no complex hyperparameter adjustment is involved, which simplifies the detection process.

[0088] In other related technologies, the continuous wavelet transform detection method is used for microsaccade detection, but the calculation time is too long, the calculation results are highly redundant, and real-time detection cannot be achieved. In addition, the computational overhead is high. In the embodiments of the present application, however, discrete wavelets can be used as a filter bank for microsaccade detection, which greatly reduces the amount of computational data, reduces the computational overhead, and significantly improves the computational efficiency, thus achieving real-time detection.

[0089] See also Figure 3 In some embodiments, obtaining the original data (i.e., 010) includes:

[0090] 011: Get the original data corresponding to the binocular;

[0091] At this point, the raw data is processed based on the filter bank to determine whether there is microsaccade movement (i.e., 020) in the coordinate sequence, including:

[0092] 021: Process the original data corresponding to the binocular through the filter bank to obtain the high-frequency component;

[0093] 022: Process the high-frequency components to determine whether there are microsaccades in the coordinate sequence.

[0094] Specifically, the eye tracking device can obtain the coordinate sequence of the left and right eye gaze points that change over time, such as Figure 4 The figure shows the coordinate sequence of the left eye's gaze point changing over time. Using the binocular raw data as the input to the filter bank, we can obtain high-frequency components. By processing these high-frequency components, we can determine whether there are microsaccades in the coordinate sequence.

[0095] See also Figure 5 and Figure 6 In some embodiments, the coordinate sequence includes a horizontal coordinate sequence. The raw data corresponding to the binocular is processed by a filter bank to obtain a high-frequency component (i.e., 021), including:

[0096] 0211: Take the horizontal coordinate sequence corresponding to the binocular eyes as the item to be transformed;

[0097] 0212: Perform discrete wavelet transform on the item to be transformed to obtain the high-frequency component.

[0098] Specifically, the coordinate sequence (x(t), y(t)) includes the horizontal coordinate sequence x(t). In the embodiment of the present application, a known wavelet function is used as a discrete wavelet basis function to obtain a discrete wavelet as a filter bank to perform discrete wavelet transform on the transform item. Discrete Wavelet Transform (DWT) is a signal processing technology that can decompose a signal into sub-bands of different frequencies and extract the time and frequency information of the signal. Each sub-band represents a different frequency range of the signal, and these sub-bands can be analyzed and processed separately to extract different features of the signal. Specifically, the Mallat fast algorithm can be used to perform discrete wavelet transform on the original data. In addition, different wavelet functions can be selected as discrete wavelet basis functions according to actual conditions. For example, Haar wavelet can be used as discrete wavelet basis function. The Haar wavelet function is shown below:

[0099]

[0100] The expression of discrete wavelet is as follows:

[0101]

[0102] in, is the wavelet function, a0 is the scale factor, b0 is the translation factor, and m and n are integers.

[0103] The horizontal coordinate sequences corresponding to the left and right eyes are respectively used as the items to be transformed, and discrete wavelet transform is performed, specifically a one-dimensional discrete wavelet transform, to obtain high-frequency components. The specific process is as follows: the items to be transformed are respectively passed through the low-pass filter h0(n) and the high-pass filter h1(n) based on the discrete wavelet basis function, and then subjected to binary frequency sampling to obtain smooth signals and difference components, respectively. The difference components are the above-mentioned high-frequency components. The high-frequency components include singular points caused by micro-saccade movements. Singular points refer to points where mutations or anomalies occur in the coordinate sequence. The obtained smooth signal is used as the new item to be transformed and iteratively processed to obtain the transformation results in the scale space and wavelet function space at different resolutions. The low-pass filter h0(n) and the high-pass filter h1(n) constitute an analysis filter bank. The smooth signal is the transformation result in the scale space, and the high-frequency component is the transformation result in the wavelet function space. The expression of the smooth signal in the i-th decomposition iteration result is as follows:

[0104]

[0105] The expression of high frequency component is as follows:

[0106]

[0107] It should be noted that the transform item can be subjected to a single one-dimensional discrete wavelet transform, or multiple one-dimensional discrete wavelet transforms can be performed on the transform item. Performing multiple one-dimensional discrete wavelet transforms yields transform results at multiple resolutions. Furthermore, both the binocular abscissa and ordinate sequences can be used as transform items for a one-dimensional discrete wavelet transform, i.e., a dual-scale one-dimensional discrete wavelet transform.

[0108] See also Figure 6 and Figure 7 In some embodiments, the high-frequency component is processed to determine whether there is microsaccade movement (i.e., 022) in the coordinate sequence, including:

[0109] 0221: Filter and reconstruct the high-frequency components to obtain a reconstructed waveform;

[0110] 0222: Post-process the reconstructed waveform and determine the singular point moments in the coordinate sequence based on the microsaccade characteristics;

[0111] 0223: Calculate the similarity of the singular point moments in the binocular coordinate sequence to determine the moment when the eye movement occurs;

[0112] 0224: Determine whether there are microsaccades in the coordinate sequence based on the time when the eye movement occurs.

[0113] Specifically, the high-frequency components in the last decomposition iteration result are first filtered, and a suitable filter can be selected according to the decomposition result to filter it to improve noise suppression. For example, when there are excessive high-frequency components in the decomposition result, the excessive high-frequency components can be removed through a threshold filter.

[0114] It should be noted that if the high-frequency component is obtained based on discrete wavelet transform, after filtering the high-frequency component and before reconstruction, interpolation processing can also be included in the high-frequency component. Specifically, quadratic interpolation can be used for interpolation processing. The high-frequency component after interpolation processing is reconstructed through the reconstruction high-pass filter g1(n) to obtain a reconstructed signal of the high-frequency component of the previous iterative signal. The expression of the i-th reconstruction iteration result (i.e., the reconstructed signal of the high-frequency component) is as follows:

[0115]

[0116] The corresponding reconstructed waveform can be obtained according to the reconstructed signal. Figure 8 As shown, Figure 8The waveform is reconstructed after a one-dimensional discrete wavelet transform. The reconstructed waveform is then post-processed to determine the singularity moments in the coordinate sequence based on the microsaccade characteristics. This post-processing can include performing local peak detection on the reconstructed waveform. In this case, a peak value greater than a predetermined threshold can be used as a microsaccade feature. In other words, the moment when the peak value exceeds the predetermined threshold is used as the singularity moment in the coordinate sequence. Figure 8 The black waveform in the middle is the reconstructed waveform of the horizontal coordinate sequence corresponding to the left eye. The points outlined by the circle are the peaks in the reconstructed waveform corresponding to the left eye that are greater than the predetermined threshold. The time positions corresponding to these multiple peaks are the singular point moments in the coordinate sequence. Figure 8 The medium gray waveform is the reconstructed waveform of the horizontal coordinate sequence corresponding to the right eye. The predetermined threshold can be set according to the actual situation. For example, 0.5 can be set as the predetermined threshold. When the absolute value of the peak is greater than 0.5, the time position corresponding to the peak is regarded as the singular point moment in the coordinate sequence.

[0117] It is understood that multiple singular moments can be detected in the coordinate series corresponding to the left and right eyes, respectively. These singular moments may include moments that are not microsaccades. Research has found that microsaccades exhibit binocular synchronization; that is, when a microsaccade occurs in the left eye, the right eye also undergoes a microsaccade. Therefore, a similarity calculation can be performed on the singular moments in the coordinate series corresponding to the two eyes to determine the time when the eye movement occurred. The time of the eye movement can then be used to determine whether microsaccades are present in the coordinate series.

[0118] See also Figure 9 In some embodiments, similarity calculation is performed on the singular point moments in the binocular coordinate sequence to determine the eye movement occurrence moment (i.e., 0223), including:

[0119] 02231: Determine whether the singular point moment in the binocular coordinate sequence is within the predetermined time window;

[0120] 02232: When the singular point moment in the binocular coordinate sequence is within a predetermined time window, determine the eye movement occurrence time according to the singular point moment in the binocular coordinate sequence.

[0121] Specifically, similarity calculation refers to the temporal similarity calculation of the singular point moments in the coordinate sequence corresponding to the two eyes. The predetermined time window can be set according to the actual situation and can be set to any value between 0-5ms. For example, the predetermined time window can be set to 3ms. If and only if the difference between a singular point moment in the coordinate sequence corresponding to the left eye and a singular point moment in the coordinate sequence corresponding to the right eye does not exceed 3ms, it can be considered that there is a micro-eye movement, and the singular point moment in the coordinate sequence corresponding to the two eyes is the moment included in the micro-eye movement. Figure 10 and Figure 11As shown in the figure, the black circle indicates the singularity moment within the predetermined time window, and the gray circle indicates the singularity moment outside the predetermined time window. After determining two singularity moments with temporal similarity in the binocular coordinate sequence, the middle moment between the two singularity moments can be taken as the eye movement occurrence time.

[0122] In related technologies, binocular correlation detection methods are used to calculate the correlation between binocular coordinate sequences to detect microsaccades. This makes it difficult to balance detection time accuracy with correlation calculation accuracy. Furthermore, the binocular correlation detection method has a fixed detection time window size and cannot adapt to microsaccades of varying durations. In the embodiments of the present application, however, a discrete wavelet transform is first performed on the raw data to determine the singularity moments. Temporal similarity calculations are then performed on these singularity moments to determine the moment when the eye movement occurred. This results in highly accurate detection results that are unaffected by the time window and can detect microsaccades of any duration.

[0123] See also Figure 12 In some embodiments, determining whether there is microsaccade (i.e., 0224) in the coordinate sequence based on the time of eye movement includes:

[0124] 02241: Extract eye movement segments within a predetermined time period before and after the eye movement occurrence moment from the coordinate sequence;

[0125] 02242: Determine whether the eye movement segment conforms to the predetermined movement trajectory;

[0126] 02243: When the eye movement segment conforms to the predetermined motion trajectory, it is determined that there is microsaccade movement in the coordinate sequence.

[0127] Specifically, the predetermined time period can be set according to the support length of the wavelet function selected above, and the predetermined time period can be set to any time period within 10-50ms. For example, the predetermined time period can be set to 15ms. Taking the moment of eye movement as the midpoint, the segment within 15ms before and after the moment of eye movement is intercepted as the eye movement segment, and the eye movement segment is presented in the gaze point motion trajectory diagram. It should be noted that the eye movement segment can be extracted from any corresponding coordinate sequence of the binocular eyes. Afterwards, it is also necessary to determine whether the eye movement segment conforms to the predetermined motion trajectory.

[0128] According to research, it is found that the trajectory of eye microsaccades is a projectile trajectory, and the direction of movement is mainly horizontal. Therefore, a predetermined micro-projectile trajectory can be set to determine whether the trajectory of the eye movement segment in the gaze point motion trajectory diagram conforms to the predetermined trajectory. When the eye movement segment does not conform to the predetermined trajectory, it is determined that the eye movement segment does not include eye microsaccades; when the eye movement segment conforms to the predetermined trajectory, it is determined that the eye movement segment includes eye microsaccades, that is, there are eye microsaccades in the coordinate sequence. Figure 13 , is the motion trajectory of the coordinate sequence corresponding to the left eye. The bold trajectory in the figure is the detected eye microsaccade motion trajectory.

[0129] The above test results can be analyzed as follows Figure 14 The velocity quantitative analysis showed that the microsaccade trajectory detected was consistent with high-speed microsaccades, with a minimum velocity greater than 0.5 ms / pixel, which, when converted to an angular coordinate system, was greater than 11.28° / s. Therefore, the detected microsaccade trajectory was considered accurate, verifying the accuracy of the test results.

[0130] It should be noted that when blinking occurs, the gaze point will move significantly. When using related technologies for detection, the data within a certain range of the blinking moment and the interval before and after will be cleared. However, the microsaccade detection method of the embodiment of the present application can effectively detect microsaccade movements before and after blinking. Figure 15 As shown in FIG, the bold track indicated by the arrow is the microsaccade motion track detected by the microsaccade detection method of the embodiment of the present application, which is not detected by the related art. This shows that the microsaccade detection method of the embodiment of the present application has a high detection accuracy.

[0131] In addition, if Figure 16 As shown, the arrow points to the microsaccade segment detected by the speed threshold detection method in the related art, which meets the speed condition of microsaccade. Figure 17 It can be seen that this segment does not meet the binocular synchronization condition and is a false detection segment of the speed threshold detection method. In the embodiment of the present application, this segment will not be detected. Therefore, it can be seen that the microsaccade detection method of the embodiment of the present application has a lower false detection rate and higher detection accuracy.

[0132] See also Figure 17 and Figure 18 In some embodiments, the near-eye display device 100 is controlled to perform corresponding operations (i.e., 030) according to the judgment result, including:

[0133] 031: When there is microsaccade movement in the coordinate sequence, obtain the microsaccade segment corresponding to the microsaccade movement;

[0134] 032: Extract microsaccade feature values ​​from microsaccade segments;

[0135] 033: Control the near-eye display device 100 to perform corresponding operations according to the micro-saccade feature value.

[0136] Specifically, the microsaccade detection method of the embodiments of the present application can be applied to the application scenario of biometric identification of the near-eye display device 100. After turning on the device, the user wears the near-eye display device 100. The near-eye display device 100 recognizes the user's wearing operation and displays a startup screen, stimulating the user to produce microsaccades. The startup screen lasts for 5-10 seconds. The eye tracker in the near-eye display device 100 continuously captures the movement trajectory of the user's eye gaze point and obtains the coordinate sequence of the eye gaze point changing over time.

[0137] When microsaccade motion is detected in the coordinate sequence, a microsaccade segment corresponding to the microsaccade motion is obtained. The multiple microsaccade segments corresponding to the multiple eye microsaccade motions can be summarized, and then the microsaccade feature values ​​in the multiple eye microsaccade segments are extracted respectively. The parameters in the microsaccade segment can be analyzed by multi-motion feature analysis to extract the microsaccade feature values. For example, the microsaccade feature values ​​can be extracted by analyzing the speed, amplitude, frequency, direction, correlation and other parameters of the microsaccade segment. The microsaccade segment can also be clustered by dimensionality reduction using an unsupervised learning method to extract the microsaccade feature values. After the microsaccade feature values ​​are extracted, the near-eye display device 100 can be controlled to perform corresponding operations according to the microsaccade feature values.

[0138] See also Figure 19 and Figure 20 In some embodiments, controlling the near-eye display device 100 to perform corresponding operations (i.e., 033) according to the micro-saccade feature value includes:

[0139] 0331: Comparing the microsaccade feature value with a predetermined feature value to determine whether the microsaccade feature value conforms to the feature value distribution of the authorized user, wherein the predetermined feature value is extracted from the eye movement segment entered by the authorized user;

[0140] 0332: When the micro-saccade feature value meets the authorized user feature value distribution, control the near-eye display device 100 to perform a screen unlocking operation;

[0141] 0333: When the micro-saccade feature value does not conform to the authorized user feature value distribution, control the near-eye display device 100 to keep the screen locked.

[0142] Specifically, the authorized user pre-records the eye movement segment, and the predetermined characteristic value is extracted from the pre-recorded eye movement segment and recorded. It can be understood that the step of extracting the predetermined characteristic value is similar to the step of extracting the micro-saccade characteristic value mentioned above, and will not be repeated here. The micro-saccade characteristic value is compared with the predetermined characteristic value to determine whether the micro-saccade characteristic value conforms to the authorized user characteristic value distribution. When the micro-saccade characteristic value conforms to the authorized user characteristic value distribution, it means that the current user is an authorized user, and the near-eye display device 100 is controlled to perform the screen unlocking operation, and the user can use the near-eye display device 100 normally. When the micro-saccade characteristic value does not conform to the authorized user characteristic value distribution, it means that the current user is not an authorized user, and the near-eye display device 100 is controlled to keep the screen locked, the user cannot use the near-eye display device 100, and the current user is prompted to check again.

[0143] It should be noted that current near-eye display devices lack a fast and effective unlocking method based on biometric technology. Since near-eye display devices block the user's facial features, facial recognition becomes more difficult, affecting the user's ability to identify and unlock the device. In addition, if the device is unlocked using fingerprints, due to the product form of the near-eye display device enclosing the human eye, it is difficult for the user to quickly find the location of the fingerprint recognition module, affecting the user experience. In the embodiment of the present application, biometric recognition is performed based on the user's eye microsaccades, which not only ensures the user experience but also improves hardware security.

[0144] See also Figure 21 and Figure 22 In some embodiments, the near-eye display device 100 is controlled to perform corresponding operations (i.e., 030) according to the judgment result, including:

[0145] 034: When microsaccades exist in the coordinate sequence, obtain the frequency, occurrence time, and / or field of view position of the microsaccades in the current screen display content scene analyzed by the near-eye display device 100;

[0146] 035: Control the near-eye display device 100 to perform corresponding operations according to the frequency, occurrence time and / or field of view position of the micro-saccade movement.

[0147] Specifically, the microsaccade detection method of the embodiment of the present application can also be applied to a recommendation algorithm based on user attention. The recommendation algorithm includes data collection and preprocessing, feature extraction and representation, similarity calculation, recommendation model, evaluation and optimization, and real-time recommendation and feedback. The user can use the near-eye display device 100 to open the application and browse the interactive program content. The eye tracker of the near-eye display device 100 captures the movement trajectory of the user's eye gaze point in real time and obtains a coordinate sequence of the eye gaze point changing over time. When microsaccade movement is detected in the coordinate sequence, the frequency, occurrence time and / or field of view position of the microsaccade movement in the current screen display content scene of the near-eye display device 100 are obtained. According to the frequency, occurrence time and / or field of view position of the microsaccade movement, the near-eye display device 100 can be controlled to perform corresponding operations.

[0148] See also Figure 23 In some embodiments, controlling the near-eye display device 100 to perform corresponding operations (i.e., 035) based on the frequency, time of occurrence, and / or field of view position of microsaccades includes:

[0149] 0351: Determine the degree to which the current screen content attracts the user's attention based on the frequency, time of occurrence, and / or visual field position of microsaccades;

[0150] 0352: when the attraction level increases, controlling the near-eye display device 100 to display screen display content associated with the current screen display content;

[0151] 0353: When the attraction level decreases, control the near-eye display device 100 to adjust the current screen display content.

[0152] Specifically, the degree to which the current screen content attracts the user's attention can be determined based on the frequency, time of occurrence, and / or visual field position of microsaccades. For example, by comparing the user's visual field position and the frequency of microsaccades before and after the current screen content appears, if the frequency decreases, it indicates that the current screen content attracts the user's attention more at that visual field position; if the frequency increases or remains unchanged, it indicates that the current screen content attracts the user less at that visual field position.

[0153] When the user's level of attraction increases, the near-eye display device 100 is controlled to display screen display content related to the current screen display content to increase user stickiness. When the user's level of attraction decreases, the near-eye display device 100 is controlled to adjust the current screen display content to increase user interest. This provides more comprehensive and detailed user usage data for the deployment of the near-eye display device 100 and more timely user experience feedback for the recommendation algorithm, thereby providing a personalized experience for users, increasing user stickiness and conversion rate, optimizing resource utilization and improving efficiency, and discovering new interests and opportunities.

[0154] See also Figure 24 The present application also provides a microsaccade detection device 200, comprising an acquisition module 10, a determination module 20, and an execution module 30. The acquisition module 10 is configured to acquire raw data, comprising a coordinate sequence of an eye's gaze point that changes over time. The determination module 20 is configured to process the raw data based on a filter bank to determine whether microsaccades are present in the coordinate sequence. The execution module 30 is configured to control the near-eye display device 100 to perform corresponding operations based on the determination result.

[0155] In some embodiments, obtaining raw data includes an acquisition module 10 specifically configured to obtain binocular raw data. In this case, a determination module 20 specifically configured to process the binocular raw data using a filter bank to obtain high-frequency components; and processing the high-frequency components to determine whether microsaccades are present in the coordinate sequence.

[0156] In some embodiments, the coordinate sequence includes a horizontal coordinate sequence. The judgment module 20 is specifically configured to use the horizontal coordinate sequences corresponding to the binoculars as items to be transformed, and perform discrete wavelet transform on the items to be transformed to obtain high-frequency components.

[0157] In some embodiments, the judgment module 20 is specifically used to filter and reconstruct the high-frequency components to obtain a reconstructed waveform; post-process the reconstructed waveform to determine the singular point moment in the coordinate sequence based on the micro-saccade characteristics; perform similarity calculation on the singular point moment in the coordinate sequence corresponding to the two eyes to determine the time when the eye movement occurs; and determine whether there is a micro-saccade movement in the coordinate sequence based on the time when the eye movement occurs.

[0158] In some embodiments, the judgment module 20 is specifically used to determine whether the singular point moment in the coordinate sequence corresponding to the binocular eyes is within a predetermined time window; when the singular point moment in the coordinate sequence corresponding to the binocular eyes is within the predetermined time window, the moment when the eye movement occurs is determined according to the singular point moment in the coordinate sequence corresponding to the binocular eyes.

[0159] In some embodiments, the judgment module 20 is specifically used to extract eye movement segments within a predetermined time period before and after the eye movement occurs from the coordinate sequence; determine whether the eye movement segments conform to the predetermined motion trajectory; when the eye movement segments conform to the predetermined motion trajectory, determine that there is micro-saccade movement in the coordinate sequence.

[0160] In some embodiments, the execution module 30 is specifically used to obtain a microsaccade segment corresponding to a microsaccade movement when there is a microsaccade movement in the coordinate sequence; extract a microsaccade feature value in the microsaccade segment; and control the near-eye display device 100 to perform corresponding operations according to the microsaccade feature value.

[0161] In some embodiments, the execution module 30 is specifically used to compare the micro-saccade feature value with a predetermined feature value to determine whether the micro-saccade feature value conforms to the authorized user feature value distribution, wherein the predetermined feature value is extracted from the eye movement segment entered by the authorized user; when the micro-saccade feature value conforms to the authorized user feature value distribution, the near-eye display device 100 is controlled to perform a screen unlocking operation; when the micro-saccade feature value does not conform to the authorized user feature value distribution, the near-eye display device 100 is controlled to keep the screen locked.

[0162] In some embodiments, the execution module 30 is specifically used to obtain the frequency, time of occurrence and / or field of view position of the micro-saccade movement in the current screen display content scene of the near-eye display device 100 when there is micro-saccade movement in the coordinate sequence; and control the near-eye display device 100 to perform corresponding operations according to the frequency, time of occurrence and / or field of view position of the micro-saccade movement.

[0163] In some embodiments, the execution module 30 is specifically used to determine the degree to which the current screen display content attracts the user's attention based on the frequency, occurrence time and / or field of view position of micro-saccade movements; when the degree of attraction increases, the near-eye display device 100 is controlled to display screen display content associated with the current screen display content; when the degree of attraction decreases, the near-eye display device 100 is controlled to adjust the current screen display content.

[0164] It should be noted that the explanation of the micro-saccade detection method in the aforementioned embodiment is also applicable to the micro-saccade detection device 200 in the embodiment of the present application, and will not be elaborated here.

[0165] See also Figure 2 The embodiment of the present application also provides a near-eye display device 100, including one or more processors 110 and a memory 120, wherein the memory 120 stores a computer program, and when the computer program is executed by the processor 110, the micro-saccade detection method of any of the above embodiments is implemented.

[0166] For example, when the computer program is executed by the processor 110, the following micro-saccade detection method is implemented:

[0167] 010: Get the original data, which includes the coordinate sequence of the eye gaze point changing over time;

[0168] 020: Process the raw data based on the filter bank to determine whether there are microsaccades in the coordinate sequence;

[0169] 030: Control the near-eye display device 100 to perform corresponding operations according to the judgment result.

[0170] For another example, when the computer program is executed by the processor 110, the following micro-saccade detection method is implemented:

[0171] 011: Get the original data corresponding to the binocular;

[0172] At this point, the raw data is processed based on the filter bank to determine whether there is microsaccade movement (i.e., 020) in the coordinate sequence, including:

[0173] 021: Process the original data corresponding to the binocular through the filter bank to obtain the high-frequency component;

[0174] 022: Process the high-frequency components to determine whether there are microsaccades in the coordinate sequence.

[0175] It should be noted that the explanations of the micro-saccade detection method and the micro-saccade detection device 200 in the aforementioned embodiment are also applicable to the near-eye display device 100 of the embodiment of the present application, and will not be elaborated here.

[0176] See also Figure 25 The present application also provides a computer-readable storage medium 300 on which a computer program 310 is stored. When the program is executed by the processor 320, the microsaccade detection method of any of the above embodiments is implemented.

[0177] For example, when the program is executed by the processor 310, the following micro-saccade detection method is implemented:

[0178] 010: Get the original data, which includes the coordinate sequence of the eye gaze point changing over time;

[0179] 020: Process the raw data based on the filter bank to determine whether there are microsaccades in the coordinate sequence;

[0180] 030: Control the near-eye display device 100 to perform corresponding operations according to the judgment result.

[0181] For another example, when the program is executed by the processor 310, the following micro-saccade detection method is implemented:

[0182] 011: Get the original data corresponding to the binocular;

[0183] At this point, the raw data is processed based on the filter bank to determine whether there is microsaccade movement (i.e., 020) in the coordinate sequence, including:

[0184] 021: Process the original data corresponding to the binocular through the filter bank to obtain the high-frequency component;

[0185] 022: Process the high-frequency components to determine whether there are microsaccades in the coordinate sequence.

[0186] It should be noted that the explanations of the microsaccade detection method and the microsaccade detection device 200 in the aforementioned embodiments are also applicable to the computer-readable storage medium 300 in the embodiments of the present application, and will not be elaborated here.

[0187] In summary, the microsaccade detection method, detection apparatus 200, near-eye display device 100, and computer-readable storage medium 300 of the embodiments of the present application acquire raw data comprising a coordinate sequence of the eye's gaze point changing over time, process the raw data using a filter bank to determine whether microsaccades are present in the coordinate sequence, and control the near-eye display device 100 to perform corresponding operations based on the determination result. This effectively avoids noise interference, improves detection accuracy, reduces computational overhead, and achieves real-time detection.

[0188] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0189] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0190] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner as necessary, and then stored in a computer memory.

[0191] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0192] Those skilled in the art will appreciate that all or part of the steps carried out in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, the various functional units in the various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk or an optical disk, etc.

[0193] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are illustrative and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. A microsaccade detection method, characterized in that: include: Acquiring raw data, wherein the raw data includes a coordinate sequence of an eye gaze point that changes over time; Processing the raw data based on a filter bank to determine whether there is microsaccade movement in the coordinate sequence; The near-eye display device is controlled to perform corresponding operations based on the judgment result.

2. The microsaccade detection method according to claim 1, wherein: The obtaining of raw data includes obtaining the raw data corresponding to the binocular vision; The processing of the raw data based on the filter bank to determine whether there is microsaccade movement in the coordinate sequence includes: Processing the original data corresponding to the binocular vision through the filter group to obtain high-frequency components; The high-frequency component is processed to determine whether microsaccades exist in the coordinate sequence.

3. The microsaccade detection method according to claim 2, wherein: The coordinate sequence includes a horizontal coordinate sequence, and the processing of the original data corresponding to the binocular eyes by the filter bank to obtain high-frequency components includes: The horizontal coordinate sequences corresponding to the binocular eyes are respectively used as items to be transformed; Perform discrete wavelet transform on the item to be transformed to obtain high-frequency components.

4. The microsaccade detection method according to claim 2, wherein: The processing of the high-frequency component to determine whether there is microsaccade movement in the coordinate sequence includes: Performing filtering and reconstruction processing on the high-frequency component to obtain a reconstructed waveform; Post-processing the reconstructed waveform to determine the singular point moment in the coordinate sequence according to micro-saccade characteristics; Performing similarity calculation on the singular point moments in the coordinate sequence corresponding to the two eyes to determine the moment when the eye movement occurs; Judging whether the microsaccade movement exists in the coordinate sequence according to the time when the eye movement occurs.

5. The microsaccade detection method according to claim 4, wherein: The similarity calculation of the singular point moments in the coordinate sequence corresponding to the two eyes to determine the moment when the eye movement occurs includes: Determine whether the singular point moment in the coordinate sequence corresponding to the binocular eyes is within a predetermined time window; When the singular point moment in the coordinate sequence corresponding to the binocular eyes is within the predetermined time window, the eye movement occurrence moment is determined according to the singular point moment in the coordinate sequence corresponding to the binocular eyes.

6. The microsaccade detection method according to claim 4, wherein: The determining whether the microsaccade movement exists in the coordinate sequence according to the eye movement occurrence time includes: Extracting eye movement segments within a predetermined time period before and after the eye movement occurrence moment from the coordinate sequence; Determining whether the eye movement segment conforms to a predetermined movement trajectory; When the eye movement segment conforms to the predetermined movement trajectory, it is determined that the microsaccade movement exists in the coordinate sequence.

7. The microsaccade detection method according to claim 1, wherein: The controlling the near-eye display device to perform corresponding operations according to the judgment result includes: When the microsaccade movement exists in the coordinate sequence, obtaining a microsaccade segment corresponding to the microsaccade movement; Extracting microsaccade feature values ​​from the microsaccade segment; The near-eye display device is controlled to perform corresponding operations according to the micro-saccade feature value.

8. The microsaccade detection method according to claim 7, wherein: The controlling the near-eye display device to perform a corresponding operation according to the micro-saccade feature value includes: Comparing the microsaccade feature value with a predetermined feature value to determine whether the microsaccade feature value conforms to a feature value distribution of an authorized user, wherein the predetermined feature value is extracted from an eye movement segment entered by the authorized user; When the micro-saccade characteristic value meets the authorized user characteristic value distribution, controlling the near-eye display device to perform a screen unlocking operation; When the micro-saccade feature value does not conform to the authorized user feature value distribution, the near-eye display device is controlled to keep locking the screen.

9. The microsaccade detection method according to claim 1, wherein: The controlling the near-eye display device to perform corresponding operations according to the judgment result includes: When the micro-saccade movement exists in the coordinate sequence, acquiring the occurrence frequency, occurrence time and / or field of view position of the micro-saccade movement in the current screen display content scene of the near-eye display device; The near-eye display device is controlled to perform corresponding operations according to the frequency, time of occurrence and / or field of view position of the micro-saccade movement.

10. The microsaccade detection method according to claim 9, wherein: The controlling the near-eye display device to perform corresponding operations according to the frequency, time of occurrence and / or field of view position of the micro-saccade movement includes: Determining the degree to which the current screen display content attracts the user's attention based on the frequency, time of occurrence, and / or visual field position of the microsaccade movement; When the attraction level increases, controlling the near-eye display device to display screen display content associated with the current screen display content; When the attraction level decreases, the near-eye display device is controlled to adjust the current screen display content.

11. A micro-saccade detection device, characterized in that: include: An acquisition module, configured to acquire raw data, wherein the raw data includes a coordinate sequence of an eye gaze point that changes over time; a judgment module, configured to process the raw data based on discrete wavelet transform to judge whether microsaccades exist in the coordinate sequence; The execution module is used to control the near-eye display device to perform corresponding operations according to the judgment result.

12. A near-eye display device, characterized in that: The near-eye display device includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the micro-saccade detection method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the micro-saccade detection method according to any one of claims 1 to 10 is implemented.

Images