A calibration method, device, terminal device and storage medium

Abstract

The application discloses a kind of calibration method, device, terminal equipment and storage medium.The method includes: the gazing behavior dataset of user is obtained when calibration substance is set to set frequency flicker, the calibration substance is the target object for calibration selected in use scene, and the gazing behavior data included in the gazing behavior dataset includes eye feature data and electroencephalogram data;According to the electroencephalogram data and the set frequency, extract effective gazing behavior dataset from the gazing behavior dataset;According to the eye feature data included in the effective behavior dataset and the position information of the calibration substance, determine calibration coefficient;According to the calibration coefficient, complete the calibration of terminal equipment.Using this method, the calibration of terminal equipment can be completed during the use of terminal equipment, and the technical effect of eye movement tracking unconscious calibration is realized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of eye tracking, and particularly relate to a calibration method and device, a terminal device and a storage medium. BACKGROUND

[0002] Eye tracking technology is a technology for estimating the coordinates of a user's gaze point position through a hardware and software system, and collecting and analyzing where people are looking at all times. It has been widely applied to scientific research experiments, VR / AR immersive interaction, commercial testing and other fields.

[0003] In the multi-point calibration process of a terminal device, such as an eye tracking device, the device needs to correctly associate the collected eye feature data with the position information of the calibration point that the user is gazing at. Therefore, the prior art requires the user to open a separate calibration interface before using the eye tracking function, guide the user to concentrate on gazing at the calibration points in turn, and complete the calibration process. If the calibration effect is poor, the user is replaced, the user moves the head significantly, the user adjusts the position of the head-mounted display, or the like during use, the calibration interface needs to be re-entered to complete the calibration, which affects the user experience. SUMMARY

[0004] Embodiments of the present application provide a calibration method, device, terminal device and storage medium, which effectively improve the efficiency of calibrating the terminal device and improve the user experience.

[0005] In a first aspect, embodiments of the present application provide a calibration method, comprising:

[0006] obtaining a gaze behavior data set of a user when a calibration object flashes at a set frequency, the calibration object being a target object selected in a use scenario for calibration, the gaze behavior data included in the gaze behavior data set including eye feature data and electroencephalogram data;

[0007] extracting an effective gaze behavior data set from the gaze behavior data set according to the electroencephalogram data and the set frequency;

[0008] determining a calibration coefficient according to the eye feature data included in the effective behavior data set and the position information of the calibration object;

[0009] completing calibration of the terminal device according to the calibration coefficient.

[0010] Optionally, the number of calibration objects is at least one, and when the number of calibration objects is at least two, the set frequencies of the calibration objects are not equal.

[0011] Optionally, when the number of calibration objects is at least two, the calibration objects are displayed simultaneously.

[0012] Optionally, the selection criteria of the calibration object satisfy at least one of the following: a display area is less than or equal to a set area threshold; a display duration is greater than or equal to a set duration; and a flickering attribute indicates that the calibration object can be flickered.

[0013] Optionally, the effective gaze behavior data set is extracted from the gaze behavior data set according to the electroencephalogram data and the set frequency, including:

[0014] The gaze behavior data set is split into gaze behavior data subsets according to a time period corresponding to the target frequency;

[0015] For each gaze behavior data subset, whether eye feature data included in the gaze behavior data subset is eye feature data of a user fixation calibration object is determined according to electroencephalogram data and electroencephalogram waveform features of the gaze behavior data subset, and if so, the gaze behavior data subset is determined as an effective gaze behavior data subset;

[0016] The electroencephalogram waveform features are waveform features of electroencephalogram signals triggered when the calibration object is flickered at the set frequency; when the calibration object is one, the target frequency is the set frequency; when the number of calibration objects is at least two, the target frequency is a set frequency with the largest value among set frequencies corresponding to the calibration objects.

[0017] Optionally, the effective gaze behavior data set is extracted from the gaze behavior data set according to the electroencephalogram data and the set frequency, including:

[0018] Eye feature data corresponding to electroencephalogram data with electroencephalogram waveform features is extracted from the gaze behavior data set;

[0019] The extracted eye feature data is determined as the effective gaze behavior data set.

[0020] Optionally, when the number of calibration objects is one, calibration of the terminal device is completed according to the calibration coefficient, including: a line-of-sight estimation algorithm model is corrected according to the calibration coefficient.

[0021] When the number of calibration objects is at least two, calibration of the terminal device is completed according to the calibration coefficient, including:

[0022] After calibration coefficients corresponding to all calibration objects are determined, a line-of-sight estimation algorithm model is corrected according to the calibration coefficients.

[0023] In a second aspect, an embodiment of the present application further provides a calibration device, including:

[0024] The acquisition module is configured to acquire a gaze behavior data set of a user when the calibration object flashes at a set frequency, the calibration object being a target object selected in a use scenario for calibration, and the gaze behavior data included in the gaze behavior data set including eye feature data and electroencephalogram data.

[0025] The extraction module is configured to extract an effective gaze behavior data set from the gaze behavior data set according to the electroencephalogram data and the set frequency.

[0026] The determination module is configured to determine a calibration coefficient according to the eye feature data included in the effective behavior data set and position information of the calibration object.

[0027] The calibration module is configured to complete calibration of the terminal device according to the calibration coefficient.

[0028] In a third aspect, an embodiment of the present application further provides a terminal device, which comprises:

[0029] one or more processors;

[0030] a storage device configured to store one or more programs;

[0031] The one or more programs are executed by the one or more processors, so that the one or more processors implement the calibration method provided by the embodiment of the present application.

[0032] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the calibration method provided by the embodiment of the present application.

[0033] The embodiment of the present application provides a calibration method, device, terminal device and storage medium. First, a calibration object is acquired to flash at a set frequency, the calibration object being a target object selected in a use scenario for calibration, and a gaze behavior data set of a user, the gaze behavior data included in the gaze behavior data set including eye feature data and electroencephalogram data. Second, an effective gaze behavior data set is extracted from the gaze behavior data set according to the electroencephalogram data and the set frequency. Third, a calibration coefficient is determined according to the eye feature data included in the effective behavior data set and position information of the calibration object. Finally, calibration of the terminal device is completed according to the calibration coefficient. By using the above technical solution, calibration of the terminal device can be completed in the use process of the terminal device, and the technical effect of unconscious calibration of eye movement tracking is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A flowchart of a calibration method provided by the embodiment of the present application;

[0035] Figure 2A flow chart of a calibration method provided for the example embodiment of the present application is shown in FIG. 1.

[0036] Figure 3 An electroencephalogram waveform diagram provided for the example embodiment of the present application is shown in FIG. 2.

[0037] Figure 4 A structure diagram of a calibration device provided for the second embodiment of the present application is shown in FIG. 3.

[0038] Figure 5 A structure diagram of a terminal device provided for the third embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0039] The present application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are intended to be merely illustrative of the present application and not in limitation thereof. It should also be noted that, for the purpose of description, only the parts related to the present application are shown in the drawings rather than all the parts.

[0040] Before the example embodiments are discussed in more detail, it should be mentioned that some of the example embodiments are described as processes or methods depicted as flow charts. Although the process is described herein as sequential process, many of the operations can be performed in parallel, concurrently or at the same time. In addition, the order of the operations can be re-arranged. The process can be terminated when its operations are completed, but can also have additional steps not included in the figure. The process can correspond to a method, function, routine, subroutine, program, etc. In addition, the embodiments and features of the embodiments in the present application can be combined with each other unless there is a conflict.

[0041] The term "comprising" and its derivations, as used in the present application, are intended to be open-ended, i.e., "including, but not limited to". The term "based on" is intended to be "based, at least in part, on". The term "one embodiment" is intended to mean "at least one embodiment".

[0042] It should be noted that the terms "first", "second", etc. mentioned in the present application are only used to distinguish the corresponding contents, and are not intended to limit the order or interdependence.

[0043] It should be noted that the modification of "one" or "multiple" mentioned in the present application is illustrative rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0044] Embodiment One

[0045] Figure 1A flowchart of a calibration method provided for the first embodiment of the present application. The method can be applied to the calibration of terminal devices. The method can be executed by a calibration device, which can be implemented by software and / or hardware and is generally integrated into a terminal device. In this embodiment, the terminal device includes, but is not limited to, a computer and a mobile phone, and other devices with eye tracking function.

[0046] The accuracy of eye tracking is an important indicator of the reliability of eye tracking function, which describes the deviation between the position coordinates of the gaze point obtained by the line-of-sight estimation algorithm and the real position coordinates of the object being gazed at. The deviation is usually represented by the angle between the two position coordinates and the line connecting the human eye.

[0047] Because eye tracking technology has very high accuracy requirements, users generally need to enter the calibration interface and complete calibration before using devices with eye tracking function. Calibration refers to the collection of eye feature data of the user while the user concentrates on gazing at one or multiple calibration points in turn. The system then correlates and calculates the "calibration point position coordinates" (i.e., the position information of the calibration object) and the "eye feature data of the user during the display of the current calibration point", and further obtains the calibration coefficient.

[0048] To ensure the accuracy of eye tracking, the system often requires the user to complete multi-point calibration. In the prior art, multiple calibration points must be displayed one by one, and the user must immediately concentrate on gazing at the calibration point as soon as it starts to display, until the calibration point stops displaying. The system continuously collects eye feature data during this process and correlates the eye feature data with the position coordinates of the currently displayed calibration point.

[0049] During the multi-point calibration process of the eye tracking system, the system needs to correctly correlate the collected eye feature data with the position coordinates of the calibration point being gazed at by the user, so that the prior art solution requires the user to open a separate calibration interface before using the eye tracking function, and guide the user to concentrate on gazing at the calibration points in turn to complete the calibration process. Once the calibration effect is poor (possibly due to the user's lack of concentration during the calibration process), the user is replaced, the user moves the head significantly (corresponding to a remote eye tracker), the user adjusts the position of the head-mounted display (corresponding to a wearable eye tracker), and other phenomena, the user often needs to exit the interface being browsed and re-enter the calibration interface to complete the calibration.

[0050] When the user is browsing content, immersed in VR\AR scene interaction, and other scenarios, the above situation will seriously affect the user's experience and increase the user's time cost when using the eye tracking function.

[0051] To solve the above technical problems, the present application provides a calibration method and a terminal device with eye tracking function. Figure 1As shown, the calibration method provided by the embodiment one of the present application comprises the following steps:

[0052] S110, acquiring the calibration object to set the frequency flicker, and obtaining the gaze behavior data set of the user.

[0053] In the embodiment, the calibration object is a target object selected in the use scenario for calibration, and the gaze behavior data included in the gaze behavior data set comprises eye feature data and electroencephalogram data.

[0054] The selection of the calibration object is not limited here, and can be determined based on the use scenario, for example, the calibration object is selected from the use scenario of the terminal device based on the display area, display time length and flicker attribute of the object in the use scenario.

[0055] For example, the object with a display area less than or equal to a set area threshold in the scenario can be selected as the calibration object. The setting of the set area threshold can be determined based on the use scenario. The smaller the display area, the higher the accuracy that can be achieved after calibration.

[0056] The embodiment does not limit the specific content of the eye feature data and the electroencephalogram data. For example, the eye feature data can include pupil position coordinates, Purkinje spot position coordinates, etc., and the electroencephalogram data can include visual evoked potential SSVEP signals and P300 evoked potential signals, etc. It should be noted that the gaze behavior data includes multiple data items, and each data item in each gaze behavior data must be synchronously acquired, that is, has the same system time stamp.

[0057] It should be noted that the number of displayed calibration objects is at least one when calibration is performed. Each calibration object corresponds to a gaze behavior data set for determining the calibration coefficient of the corresponding calibration object. Each gaze behavior data set can include multiple gaze behavior data, for example, a set formed by all gaze behavior data of a calibration object from display to stop display is determined as a gaze behavior data set. The electroencephalogram data in the present application is used to filter the eye feature data of the user when not gazing at the calibration object in the gaze behavior data set, thereby improving the accuracy of the calibration coefficient.

[0058] In the embodiment, the calibration object flickers at a set frequency in the use scenario, so that the electroencephalogram signal can be triggered when the user gazes at the calibration object.

[0059] This step acquires the gaze behavior data set of the user in the calibration object display process, and the corresponding eye feature data can be determined through the electroencephalogram data in the gaze behavior data set whether it is the data acquired when the user gazes at the calibration object, thereby improving the calibration efficiency.

[0060] The set frequency is not limited in the embodiment, and can be determined based on the use scenario.

[0061] The step does not limit the technical means for obtaining the gaze behavior data set. For example, different data corresponds to different collectors. The step can obtain the corresponding data collected by the collector to form the gaze behavior data set.

[0062] In one embodiment, the number of the calibration objects is at least one. When the number of the calibration objects is at least two, the set frequencies corresponding to the calibration objects are different.

[0063] In the embodiment, the terminal device uses a page that can include multiple calibration objects. The set frequencies of the calibration objects are different, which are used to distinguish different calibration objects.

[0064] When the number of the calibration objects in a display page is at least two, the calibration objects can flash at the same set frequency or can flash at different or same set frequencies in sequence.

[0065] In one embodiment, when the number of the calibration objects is at least two, the calibration objects are displayed simultaneously.

[0066] Each calibration object can flash at different set frequencies.

[0067] In one embodiment, the selection criteria of the calibration objects satisfy at least one of the following conditions: the display area is less than or equal to a set area threshold; the display duration is greater than or equal to a set duration; and the flashing attribute can be flashed.

[0068] The display area can be considered as the area of the object displayed in the display page in the use scenario. The display duration can be considered as the duration of the object continuously displayed in the use scenario. The flashing attribute can be an attribute representing whether the object in the use scenario can flash. The determination of the flashing attribute is based on the use scenario, for example, based on the use scenario to determine whether the object can flash. The specific values of the area threshold and the set duration are not limited.

[0069] S120, extracting an effective gaze behavior data set from the gaze behavior data set according to the electroencephalogram data and the set frequency.

[0070] The effective gaze behavior data set can be considered as the collected data when the user gazes at the calibration point, for example, the collected electroencephalogram data and the collected eye feature data.

[0071] The embodiment can determine the corresponding time period based on the set frequency, and then determine the brain wave waveform characteristics that the electroencephalogram data should have when the user gazes at the calibration object based on the time period. The electroencephalogram data is filtered based on the determined brain wave waveform characteristics to realize the screening of the effective gaze behavior data set. The brain wave waveform characteristics include but are not limited to: latency duration range, peak range of wave crest and trough, etc.

[0072] In one embodiment, when extracting the effective gaze behavior data, this step can directly analyze all the electroencephalogram data in the gaze behavior data set to extract the eye features corresponding to the electroencephalogram data that meet the brain wave waveform characteristics, for forming the effective gaze behavior data set.

[0073] In one embodiment, when extracting the effective gaze behavior data, this step can first split the gaze behavior data set to obtain a plurality of gaze behavior data subsets, and then analyze each gaze behavior data subset to obtain the effective gaze behavior data set. The splitting standard is not limited here, such as based on the calibration object flickering time period as the splitting standard. For example, a gaze behavior data subset is formed by setting a number of time periods of gaze behavior data. The value of the number is not limited here.

[0074] S130, determining the calibration coefficient according to the eye feature data included in the effective behavior data set and the position information of the calibration object.

[0075] This step does not limit the specific technical means for determining the calibration coefficient, such as determining the gaze point position information of the user based on the eye feature data, such as the gaze point position coordinates. Then determine the calibration coefficient based on the gaze point position information and the position information of the calibration point.

[0076] S140, completing the calibration of the terminal device according to the calibration coefficient.

[0077] The technical means for completing the calibration of the terminal device based on the calibration coefficient is not limited.

[0078] The calibration method provided by the embodiment one of the application first acquires a calibration object and a user's gaze behavior data set when the calibration object flashes at a set frequency, the calibration object is a target object selected in a use scenario for calibration, the gaze behavior data included in the gaze behavior data set includes eye feature data and electroencephalogram data; secondly, valid gaze behavior data set is extracted from the gaze behavior data set according to the electroencephalogram data and the set frequency; then, a calibration coefficient is determined according to the eye feature data included in the valid behavior data set and position information of the calibration object; finally, calibration of a terminal device is completed according to the calibration coefficient. By selecting the calibration object in the use scenario to complete the calibration, the calibration of the terminal device can be completed in the use process of the terminal device, and the technical effect of unconscious calibration of eye movement tracking is realized.

[0079] On the basis of the above-mentioned embodiments, variant embodiments of the above-mentioned embodiments are proposed, and it should be noted that, in order to make the description brief, only the differences from the above-mentioned embodiments are described in the variant embodiments.

[0080] In one embodiment, the valid gaze behavior data set is extracted from the gaze behavior data set according to the electroencephalogram data and the set frequency, including:

[0081] The gaze behavior data set is split to obtain gaze behavior data subsets according to a time period corresponding to the target frequency;

[0082] For each gaze behavior data subset, whether the eye feature data included in the gaze behavior data subset is the eye feature data of the user gazing at the calibration object is determined according to the electroencephalogram data and the electroencephalogram waveform feature of the gaze behavior data subset, and if yes, the gaze behavior data subset is determined as a valid gaze behavior data subset;

[0083] The electroencephalogram waveform feature is the waveform feature of the electroencephalogram signal triggered when the calibration object flashes at the set frequency; when the calibration object is one, the target frequency is the set frequency; when the number of the calibration objects is at least two, the target frequency is the set frequency with the maximum value among the set frequencies corresponding to each calibration object.

[0084] Splitting the gaze behavior data set and then analyzing each gaze behavior data subset can accelerate the processing speed.

[0085] When whether the eye feature data corresponding to the electroencephalogram data in each gaze behavior data subset is the eye feature data of the user gazing at the calibration object is determined based on the waveform feature corresponding to the electroencephalogram data, when the electroencephalogram data has the electroencephalogram waveform feature, the eye feature data corresponding to the electroencephalogram data is the eye feature data collected when the user gazes at the calibration object, otherwise, the eye feature data collected when the user does not gaze at the calibration object.

[0086] In one embodiment, according to the brain electrical data and the set frequency, an effective gaze behavior data set is extracted from the gaze behavior data set, comprising:

[0087] The eye feature data corresponding to the brain electrical data with the brain electrical waveform feature is extracted from the gaze behavior data set.

[0088] The extracted eye feature data is determined as the effective gaze behavior data set.

[0089] The embodiment can directly analyze the gaze behavior data set to screen out the effective gaze behavior data set.

[0090] In one embodiment, when the number of the calibration objects is one, the calibration of the terminal device is completed according to the calibration coefficient, comprising: correcting the line-of-sight estimation algorithm model according to the calibration coefficient.

[0091] When the number of the calibration objects is at least two, the calibration of the terminal device is completed according to the calibration coefficient, comprising:

[0092] After the calibration coefficients corresponding to all the calibration objects are determined, the line-of-sight estimation algorithm model is corrected according to the calibration coefficients.

[0093] The line-of-sight estimation algorithm model can be considered as a model for line-of-sight estimation. After the calibration coefficient is determined, the line-of-sight trajectory algorithm model is corrected according to all the calibration coefficients to complete the calibration of the terminal device. When the calibration object is one, the terminal device is calibrated directly after the calibration coefficient is determined. When the calibration object is multiple, the terminal device is calibrated after the calibration coefficient corresponding to each calibration object is determined.

[0094] It should be noted that the calibration object in the application can be considered as a calibration point. The difference lies in that the calibration object in the application is an object that should exist in the use scene of the terminal device. The application does not need to set a calibration interface, and the calibration can be completed directly in the use process of the terminal device.

[0095] The calibration method provided by the example embodiments of the application can be considered as an eye movement tracking unconscious calibration method. The technical principle of the calibration method provided by the application includes that when a user watches an object that flashes at a fixed frequency, the brain wave signal will present regular fluctuations, so the object coordinates that the user is gazing at can be determined to calibrate the terminal device.

[0096] The eye movement tracking unconscious calibration system is composed of a data acquisition module (i.e., an acquisition module), a data screening module (i.e., an extraction module), and a data calculation module (i.e., a calibration module).

[0097] 1. Data acquisition module, which can automatically select calibration objects, i.e. calibration objects, in the scene according to the preset selection requirements, and collect the user's gaze behavior data while the calibration objects are flashing at different frequencies (set frequencies). The gaze behavior data includes eye feature data and electroencephalogram data.

[0098] 2. Data filtering module, which can determine which calibration object the user is gazing at according to the electroencephalogram data, and then integrate the effective gaze behavior data set.

[0099] 3. Data calculation module, which can associate the eye feature data in the effective gaze behavior data set with the calibration point position coordinates to calculate the calibration coefficient.

[0100] Figure 2 A flowchart of a calibration method provided for the example embodiment of the present application is shown in Figure 2 , the eye tracking unconscious calibration method, which consists of the following steps:

[0101] Preset calibration object selection requirements.

[0102] The user directly enters the use scene. When the eye tracking system calibration starts, the system automatically selects multiple calibration objects, confirms the calibration point position coordinates and collects the user's gaze behavior data. The calibration objects flash at different frequencies. The calibration point position coordinates are confirmed according to the geometric center of the calibration objects. The gaze behavior data includes eye feature data and electroencephalogram data.

[0103] Divide the gaze behavior data into several gaze behavior data packets.

[0104] According to the electroencephalogram data in the gaze behavior data packet, determine whether the eye feature data contained in each gaze behavior data packet is the eye feature data of the user gazing at the calibration object. If yes, further determine which calibration object the user is gazing at; if no, the gaze behavior data packet is invalid.

[0105] Integrate the eye feature data of the user gazing at a certain calibration object into an effective gaze behavior data set.

[0106] Associate the eye feature data in the effective gaze behavior data set with the calibration point position coordinates to calculate the calibration coefficient.

[0107] Based on all the calibration coefficients, complete the modification of the line-of-sight estimation algorithm model. Calibration is complete.

[0108] In this example, the calibration point can be considered as the calibration object.

[0109] The eye tracking unconscious calibration method is described in detail as follows:

[0110] Step one: preset the selection requirement of the calibration object, the selection requirement can be the threshold range of the object display area.

[0111] The threshold range of the calibration object display area can be determined based on the display area of the terminal device, for example, in the case of a 1920*1080 resolution display, when the scene has a high requirement for eye tracking accuracy, the threshold can be set to 100-200 pixels; when the scene has a low requirement for eye tracking accuracy, the threshold can be set to 500-2000 pixels.

[0112] Step two: the system directly displays the scene content that the user needs to browse. When the eye tracking system calibration starts, the system automatically selects multiple calibration objects in the scene content according to the preset selection requirement. And select the geometric center of each calibration object as the calibration point position coordinate. The geometric center of the calibration object can be defined as the intersection point between the longest axis (the line connecting the pair of points with the longest distance on the edge) and the shortest axis of the object outline.

[0113] Step three: the calibration object starts to flash at different frequencies, and at the same time the system starts to collect the user's gaze behavior data. The gaze behavior data includes eye feature data and electroencephalogram data.

[0114] Step four: among all the flashing calibration objects, take the fastest flashing frequency as the standard to divide the gaze behavior data into several gaze behavior data packets.

[0115] For example, the flashing frequency is 5Hz, 3Hz, 2Hz, then take 5Hz as the standard, and the collected gaze behavior data every 200ms as a gaze behavior data packet.

[0116] Step five: according to the electroencephalogram data in the gaze behavior data packet, judge whether the eye feature data contained in each gaze behavior data packet is the eye feature data of the user gazing at the calibration object. If yes, further confirm which calibration object the user is gazing at; if not, the gaze behavior data packet is invalid.

[0117] For example, based on the flashing frequency of each calibration object, determine the electroencephalogram waveform characteristics of the SSVEP signal or P300 signal that the user should produce when gazing at a certain calibration object. The electroencephalogram waveform characteristics can be the latency duration range, the peak value range of the wave crest and trough, etc. If the electroencephalogram data in the gaze behavior data packet has the electroencephalogram waveform characteristics of gazing at a certain calibration object, it is considered that the eye feature data in the gaze behavior data packet is the eye feature data of the user gazing at the calibration object; otherwise, it is not the eye feature data of the user gazing at the calibration object, and is invalid data.

[0118] Figure 3A brain wave form diagram is provided for the example embodiment of the present application, taking the P300 signal collected on the Pz electrode (midline electrode) on the user's head as an example, the flicker frequency of a certain calibration point is 2Hz, i.e. flickering once every 500ms, forming a visual stimulus for the user, when the peak value of the P300 signal in the brain wave form (brain wave waveform) diagram is 6.7μV and appears once every 500ms at the same frequency, it can be determined that the current user is gazing at the first calibration point. As shown in Figure 3 the P300 signal peak value from 300ms to 1800ms in the diagram is the same as the calibration point flicker, so the eye feature data in the gaze behavior data packet within these 1500ms is the eye feature data of the user gazing at the calibration object.

[0119] Step six: the eye feature data of the user gazing at a certain calibration object is integrated into an effective gaze behavior data set.

[0120] For example, the eye feature data of the user gazing at calibration object A is integrated into effective gaze behavior data set A.

[0121] Step seven: associate the eye feature data in the effective gaze behavior data set with the calibration point position coordinates of the calibration object, and calculate the calibration coefficient.

[0122] For example, the gaze estimation algorithm reads the pupil position coordinates and Purkinje spot position coordinates contained in the eye feature data set A, and calculates the gaze point position coordinates of the current user. Further, based on the deviation value of the gaze point position coordinates and the calibration point position coordinates of calibration object A, the gaze estimation algorithm model is corrected, thereby achieving the purpose of rectification, and the correction coefficient, i.e. the calibration coefficient A.

[0123] Step eight: based on all the calibration coefficients, the gaze estimation algorithm model is corrected. Calibration is completed.

[0124] Before using the eye movement tracking function, the user does not need to complete calibration in a separate calibration interface, but can directly enter the use scene to complete "invisible" calibration in an unconscious state. Once the user needs to be recalibrated, the user does not need to exit the interface being browsed, but can complete calibration directly in the current interface. This significantly improves the user's experience and reduces the user's time cost when using the eye movement tracking function.

[0125] In addition to the embodiment of step five in the above-mentioned eye movement tracking unconscious calibration method, which collects P300 signals on the Pz electrode (midline electrode) on the user's head, another embodiment based on the present solution can be provided.

[0126] The SSVEP signal is collected on Fz (midline of forehead electrode), O1 (left occipital electrode), Oz (midline of occipital electrode), and O2 (right occipital electrode), the SSVEP signal is extracted by a canonical correlation analysis (CCA) algorithm, a linear combination of the SSVEP signals of multiple channels with the maximum correlation coefficient is calculated, and then the frequency of the SSVEP signal is obtained based on the recognition of the peak value of the SSVEP signal and the maximum correlation coefficient by the canonical correlation analysis algorithm.

[0127] Further, the frequency of the SSVEP signal in the gaze behavior data packet is compared with the flicker frequency of the calibration object, and the eye feature data in the gaze behavior data packet with the same frequency of the SSVEP signal and the calibration object is the eye feature data of the user gazing at the calibration object.

[0128] Embodiment Two

[0129] Figure 4 A structural schematic diagram of a calibration device provided for the second embodiment of the present application, which can be applicable to the calibration of a terminal device, wherein the device can be realized by software and / or hardware, and is generally integrated on the terminal device.

[0130] As shown in Figure 4 , the device comprises:

[0131] The acquisition module 31 is configured to acquire a gaze behavior dataset of a user when the calibration object is flickered at a set frequency, the calibration object being a target object selected for calibration in a use scenario, and the gaze behavior data included in the gaze behavior dataset including eye feature data and electroencephalogram data.

[0132] The extraction module 32 is configured to extract an effective gaze behavior dataset from the gaze behavior dataset according to the electroencephalogram data and the set frequency.

[0133] The determination module 33 is configured to determine a calibration coefficient according to the eye feature data included in the effective behavior dataset and the position information of the calibration object.

[0134] The calibration module 34 is configured to complete the calibration of the terminal device according to the calibration coefficient.

[0135] In the embodiment, the device first acquires the calibration object to set the frequency flicker of the user's gaze behavior data set, the calibration object is the target object selected in the use scene for calibration, the gaze behavior data included in the gaze behavior data set includes eye feature data and electroencephalogram data; secondly, the effective gaze behavior data set is extracted from the gaze behavior data set according to the electroencephalogram data and the set frequency by the extraction module 32; then the calibration coefficient is determined according to the eye feature data included in the effective behavior data set and the position information of the calibration object by the determination module 33; finally, the calibration of the terminal device is completed according to the calibration coefficient by the calibration module 34.

[0136] The embodiment provides a calibration device, which can complete the calibration of the terminal device in the use process of the terminal device by selecting the calibration object in the use scene to complete the calibration, and realizes the technical effect of unconscious calibration of eye movement tracking.

[0137] In one embodiment, the number of calibration objects is at least one, and when the number of calibration objects is at least two, the set frequencies corresponding to each calibration object are not equal.

[0138] In one embodiment, when the number of calibration objects is at least two, each calibration object is displayed at the same time.

[0139] In one embodiment, the selection criteria of the calibration object satisfy at least one of the following: the display area is less than or equal to a set area threshold; the display duration is greater than or equal to a set duration; and the flicker attribute indication can be flickered.

[0140] In one embodiment, the extraction module 32 is specifically used for:

[0141] According to the time period corresponding to the target frequency, the gaze behavior data set is split to obtain a gaze behavior data subset;

[0142] For each gaze behavior data subset, according to the electroencephalogram data and the electroencephalogram waveform feature of the gaze behavior data subset, it is determined whether the eye feature data included in the gaze behavior data subset is the eye feature data of the user gazing at the calibration object, and if so, the gaze behavior data subset is determined as an effective gaze behavior data subset;

[0143] Wherein, the electroencephalogram waveform feature is the waveform feature of the electroencephalogram signal triggered when the calibration object flickers at the set frequency; when the calibration object is one, the target frequency is the set frequency; when the number of calibration objects is at least two, the target frequency is the set frequency with the maximum value among the set frequencies corresponding to each calibration object.

[0144] In one embodiment, the extraction module 32 is specifically used for:

[0145] extracting eye feature data corresponding to electroencephalogram data with electroencephalogram waveform features from the gaze behavior data set;

[0146] determining the extracted eye feature data as valid gaze behavior data set.

[0147] In one embodiment, when the number of calibration objects is one, the calibration of the terminal device is completed according to the calibration coefficient, including: correcting the line-of-sight estimation algorithm model according to the calibration coefficient.

[0148] When the number of calibration objects is at least two, the calibration of the terminal device is completed according to the calibration coefficient, including:

[0149] After the calibration coefficients corresponding to all calibration objects are determined, the line-of-sight estimation algorithm model is corrected according to each calibration coefficient.

[0150] The calibration device described above can perform the calibration method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0151] Embodiment three

[0152] Figure 5 A structural schematic diagram of a terminal device provided by Embodiment three of the present application is shown in FIG. 3. As shown in FIG. 3, the terminal device provided by Embodiment four of the present application includes one or more processors 41 and a storage device 42. Figure 5 The processor 41 in the terminal device can be one or more, and the storage device 42 is used for storing one or more programs; the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 implement the calibration method described in any of the embodiments of the present application. Figure 5 The processor 41 in the terminal device can be one or more, and the storage device 42 is used for storing one or more programs; the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 implement the calibration method described in any of the embodiments of the present application.

[0153] The terminal device can further include an input device 43 and an output device 44.

[0154] The processor 41, the storage device 42, the input device 43 and the output device 44 in the terminal device can be connected through a bus or other means, Figure 5 for example, through a bus.

[0155] The storage device 42 in the terminal device, as a kind of computer readable storage medium, can be used to store one or more programs, and the program can be a software program, a computer executable program and a module, such as the program instruction / module corresponding to the calibration method provided by Embodiment one of the present application (for example, the program instruction / module for correcting the line-of-sight estimation algorithm model according to the calibration coefficient). Figure 4The modules in the calibration device shown include: an acquisition module 31, an extraction module 32, a determination module 33, and a calibration module 34. The processor 41 executes various functional applications and data processing of the terminal device by running the software programs, instructions, and modules stored in the storage device 42, i.e., implements the calibration method in the above method embodiments.

[0156] The storage device 42 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the storage device 42 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some examples, the storage device 42 can further include a memory remotely arranged with respect to the processor 41, which can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0157] The input device 43 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function control of the terminal device. The output device 44 can include a display device such as a display screen.

[0158] And when the above terminal device includes one or more programs executed by the one or more processors 41, the programs perform the following operations:

[0159] Acquire a calibration object to set the frequency flicker, the calibration object is a target object selected for calibration in a use scenario, the gaze behavior data set includes eye feature data and electroencephalogram data;

[0160] According to the electroencephalogram data and the set frequency, extract an effective gaze behavior data set from the gaze behavior data set;

[0161] According to the eye feature data included in the effective behavior data set and the position information of the calibration object, determine a calibration coefficient;

[0162] According to the calibration coefficient, complete the calibration of the terminal device.

[0163] Embodiment five

[0164] Embodiment five of the present application provides a computer readable storage medium, which stores a computer program, the program is executed by a processor to execute a calibration method, the method comprises:

[0165] Obtaining a calibration object and a user's gaze behavior data set when a frequency flicker is set, the calibration object being a target selected in a use scenario for calibration, the gaze behavior data set including gaze behavior data including eye feature data and electroencephalogram data;

[0166] Extracting an effective gaze behavior data set from the gaze behavior data set according to the electroencephalogram data and the set frequency;

[0167] Determining a calibration coefficient according to eye feature data included in the effective behavior data set and position information of the calibration object;

[0168] Completing calibration of a terminal device according to the calibration coefficient.

[0169] Optionally, the program, when executed by the processor, can further be used to execute the calibration method provided by any of the embodiments of the present application.

[0170] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0171] The computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, in which a computer readable program code is carried. Such a propagated data signal can take multiple forms, including but not limited to: an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can be used to carry or store a program for use by or in connection with an instruction execution system, device or component.

[0172] The program code embodied on the computer readable media can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, Radio Frequency (RF) etc., or any suitable combination of the foregoing.

[0173] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0174] Note that, as previously discussed, the above are merely preferred embodiments of the present application and the principles of technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, reconfigurations and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A calibration method, characterized in that, include: Acquire a user's gaze behavior dataset when a calibration object flashes at a set frequency. The calibration object is a target object selected in the usage scenario for calibration. The gaze behavior dataset includes eye feature data and electroencephalogram (EEG) data. The corresponding time period is determined based on the set frequency, and the brainwave waveform characteristics are determined based on the time period. The brainwave waveform characteristics are the waveform characteristics of the brainwave signal triggered when the calibrator flashes at the set frequency. Based on the EEG data and the EEG waveform features corresponding to the set frequency, a valid gaze behavior dataset is extracted from the gaze behavior dataset, and the EEG waveform features are used to filter the EEG data. The calibration coefficients are determined based on the eye feature data included in the effective behavior dataset and the location information of the calibration object; The terminal device is calibrated according to the calibration coefficients.

2. The method according to claim 1, characterized in that, The number of calibrators is at least one, and when the number of calibrators is at least two, the set frequencies corresponding to each calibrator are not equal.

3. The method according to claim 1, characterized in that, When there are at least two calibrators, each calibrator is displayed simultaneously.

4. The method according to claim 1, characterized in that, The selection criteria for the calibrator must meet at least one of the following: the display area is less than or equal to a set area threshold; the display duration is greater than or equal to a set duration; and the blink attribute indicator can be blinked.

5. The method according to claim 1, characterized in that, Based on the EEG data and the EEG waveform characteristics corresponding to the set frequency, a valid gaze behavior dataset is extracted from the gaze behavior dataset, including: Based on the time period corresponding to the target frequency, the gaze behavior dataset is split to obtain a gaze behavior data subset; For each subset of gaze behavior data, based on the EEG data and EEG waveform characteristics of the subset of gaze behavior data, it is determined whether the eye feature data included in the subset of gaze behavior data is the eye feature data of the user gaze calibrator. If so, the subset of gaze behavior data is determined as a valid subset of gaze behavior data. When there is one calibrator, the target frequency is the set frequency; when there are at least two calibrators, the target frequency is the set frequency with the largest value among the set frequencies corresponding to each calibrator.

6. The method according to claim 1, characterized in that, Based on the EEG data and the EEG waveform characteristics corresponding to the set frequency, a valid gaze behavior dataset is extracted from the gaze behavior dataset, including: Extract eye feature data corresponding to EEG data with EEG waveform features from the gaze behavior dataset; The extracted eye feature data were identified as the effective gaze behavior dataset.

7. The method according to claim 1, characterized in that, When the number of calibrators is one, the calibration of the terminal device is completed according to the calibration coefficient, including: correcting the line-of-sight estimation algorithm model according to the calibration coefficient; When the number of calibrators is at least two, the calibration of the terminal device is completed according to the calibration coefficient, including: After the calibration coefficients for all calibrators have been determined, the line-of-sight estimation algorithm model is corrected based on the calibration coefficients.

8. A calibration device, characterized in that, include: The acquisition module is used to acquire a user's gaze behavior dataset when the calibration object flashes at a set frequency. The calibration object is a target object selected in the usage scenario for calibration. The gaze behavior dataset includes eye feature data and electroencephalogram (EEG) data. The module determines the corresponding time period based on the set frequency and determines the EEG waveform characteristics based on the time period. The EEG waveform characteristics are the waveform characteristics of the EEG signal triggered when the calibration object flashes at the set frequency. The extraction module is used to extract a valid gaze behavior dataset from the gaze behavior dataset based on the EEG data and the EEG waveform features corresponding to the set frequency. The determination module is used to determine the calibration coefficient based on the eye feature data included in the effective behavior dataset and the position information of the calibration object; The calibration module is used to calibrate the terminal device according to the calibration coefficient.

9. A terminal device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the calibration method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the calibration method as described in any one of claims 1-7.

Images