A reading-based eye movement device calibration method, system, and related devices

Abstract

The application provides a reading-based eye movement device calibration method and system and related devices, collects a set of uncalibrated eye movement coordinates of a user when the user wears an eye movement device to read a specified text, obtains a set of text unit coordinates and a set of edge unit coordinates; calculates a gaze density of the eye movement coordinate points; performs center alignment on the eye movement coordinate points and the text unit coordinates included in the set of text unit coordinates to obtain a first calibration matrix; sequentially performs a point matching process, a weight calculation process and a gradient descent process and repeats the processes for multiple rounds to obtain multiple second calibration matrices; and calculates a final calibration matrix for calibrating the eye movement device by using the first calibration matrix and the multiple second calibration matrices. The scheme finds a corresponding relationship between the gaze behavior and the text content from the reading behavior of the user using the computer to complete the implicit calibration of the eye movement device, simplifies the calibration process of the eye movement device, and does not affect the normal use of the computer by the user during calibration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a reading-based eye movement device calibration method and system and related devices. BACKGROUND

[0002] Currently, eye movement devices (various types of eye trackers) usually use explicit calibration to calibrate, specifically, the user first gazes at multiple points on the screen and obtains the gaze vectors of the user when gazing at these points, and then solves the necessary parameters in the eye movement estimation process through the obtained vectors, thereby completing the calibration of the eye movement device.

[0003] However, the calibration process of the above-mentioned explicit calibration method is relatively cumbersome and will interfere with the user's normal use of the computer. SUMMARY

[0004] Therefore, the embodiments of the present application provide a reading-based eye movement device calibration method, system and related devices to simplify the calibration process of the eye movement device and avoid interfering with the user's normal use of the computer during calibration.

[0005] To achieve the above object, the embodiments of the present application provide the following technical solutions:

[0006] The first aspect of the embodiments of the present application discloses a reading-based eye movement device calibration method, which comprises:

[0007] Collecting a set of uncalibrated eye movement coordinates of a user when reading a specified text while wearing an eye movement device, and obtaining a set of text unit coordinates and a set of edge unit coordinates;

[0008] Calculating the gaze density of the eye movement coordinate points contained in the set of uncalibrated eye movement coordinates;

[0009] Centering the eye movement coordinate points and the text unit coordinates contained in the set of text unit coordinates to obtain a first calibration matrix;

[0010] According to the set of uncalibrated eye movement coordinates, the set of text unit coordinates, the set of edge unit coordinates and the gaze density, sequentially performing a point matching process, a weight calculation process and a gradient descent process and repeating multiple rounds to obtain multiple second calibration matrices;

[0011] Using the first calibration matrix and multiple second calibration matrices to calculate a final calibration matrix for calibrating the eye movement device.

[0012] Preferably, the process of performing the point matching process comprises:

[0013] matching the eye movement coordinate points included in the uncalibrated eye movement coordinate set respectively with the text unit coordinates included in the text unit coordinate set and the edge unit coordinates included in the edge unit coordinate set to obtain a set of point pairs;

[0014] wherein the set of point pairs includes multiple groups of point pairs, and each group of point pairs includes the eye movement coordinate point and the text unit coordinate that are closest to each other in the same line, or the eye movement coordinate point and the edge unit coordinate that are closest to each other.

[0015] Preferably, the process of performing the weight calculation procedure includes:

[0016] determining the weight of each group of point pairs in the set of point pairs.

[0017] Preferably, the process of determining the weight of each group of point pairs in the set of point pairs includes:

[0018] for each group of point pairs in the set of point pairs, if the point pair includes the eye movement coordinate point and the text unit coordinate, and the text content corresponding to the text unit coordinate included in the point pair is not a punctuation, calculating the gaze time estimate of the text unit coordinate included in the point pair;

[0019] using the gaze density of the eye movement coordinate point included in the point pair and the gaze time estimate of the text unit coordinate to calculate the weight of the point pair;

[0020] if the point pair includes the eye movement coordinate point and the text unit coordinate, and the text content corresponding to the text unit coordinate included in the point pair is a punctuation, determining the weight of the point pair to be a first constant, and the first constant is a negative number;

[0021] if the point pair includes the eye movement coordinate point and the edge unit coordinate, determining the weight of the point pair to be a second constant, and the second constant is a negative number.

[0022] Preferably, the process of performing the gradient descent procedure includes:

[0023] performing gradient descent solving according to the eye movement coordinate points, the text unit coordinates, the edge unit coordinates, and the weights of the point pairs, with the weighted distance from the eye movement coordinate points to the text unit coordinates or the edge unit coordinates after moving by a second calibration matrix being minimized as the optimization target, to obtain the second calibration matrix;

[0024] applying the second calibration matrix to the eye movement coordinate points.

[0025] Preferably, the final calibration matrix for calibrating the eye movement device is calculated by using the first calibration matrix and the plurality of second calibration matrices, including:

[0026] The plurality of second calibration matrices obtained by sequentially performing the point matching process, the weight calculation process and the gradient descent process in the Nth round to the Mth round are averaged to obtain a third calibration matrix, and M is greater than N.

[0027] The product of the first calibration matrix and the third calibration matrix is calculated to obtain the final calibration matrix for calibrating the eye movement device.

[0028] The second aspect of the embodiment of the application discloses a reading-based eye movement device calibration system, and the system comprises:

[0029] The acquisition module is configured to acquire an uncalibrated eye movement coordinate set of a user when the user reads a specified text while wearing an eye movement device, and obtain a text unit coordinate set and an edge unit coordinate set.

[0030] The first calculation module is configured to calculate the gaze density of an eye movement coordinate point included in the uncalibrated eye movement coordinate set.

[0031] The alignment module is configured to perform center alignment on the eye movement coordinate point and a text unit coordinate included in the text unit coordinate set to obtain a first calibration matrix.

[0032] The execution module is configured to sequentially perform a point matching process, a weight calculation process and a gradient descent process and repeat the processes for multiple rounds according to the uncalibrated eye movement coordinate set, the text unit coordinate set, the edge unit coordinate set and the gaze density, and obtain a plurality of second calibration matrices.

[0033] The second calculation module is configured to calculate a final calibration matrix for calibrating the eye movement device by using the first calibration matrix and the plurality of second calibration matrices.

[0034] Preferably, the execution module performs the point matching process, including:

[0035] The eye movement coordinate point included in the uncalibrated eye movement coordinate set is matched with the text unit coordinate included in the text unit coordinate set and the edge unit coordinate included in the edge unit coordinate set, respectively, to obtain a point pair set.

[0036] The point pair set includes a plurality of point pairs, and each point pair includes the eye movement coordinate point and the text unit coordinate that belong to the same line and are closest to each other, or each point pair includes the eye movement coordinate point and the edge unit coordinate that are closest to each other.

[0037] The third aspect of the embodiment of the present application discloses an electronic device, comprising a memory and a processor;

[0038] The memory is used for storing a computer program;

[0039] The processor is used for executing the computer program, and specifically used for implementing the reading-based eye movement device calibration method disclosed in the first aspect of the embodiment of the present application.

[0040] The fourth aspect of the embodiment of the present application discloses a computer storage medium used for storing a computer program, and the computer program is executed to specifically implement the reading-based eye movement device calibration method disclosed in the first aspect of the embodiment of the present application.

[0041] Based on the reading-based eye movement device calibration method, system and related device provided by the above embodiment of the present application, the method is as follows: a set of uncalibrated eye movement coordinates of a user when reading a specified text while wearing an eye movement device is collected, and a set of text unit coordinates and a set of edge unit coordinates are obtained; a gaze density of an eye movement coordinate point contained in the set of uncalibrated eye movement coordinates is calculated; the eye movement coordinate point is centrally aligned with the text unit coordinates contained in the set of text unit coordinates to obtain a first calibration matrix; a point matching process, a weight calculation process and a gradient descent process are sequentially executed and repeated for multiple rounds to obtain multiple second calibration matrices; and the first calibration matrix and the multiple second calibration matrices are used to calculate a final calibration matrix for calibrating the eye movement device. The scheme finds a corresponding relationship between a gaze behavior and a text content from a reading behavior of the user using a computer, thereby establishing an implicit relationship and completing implicit calibration of the eye movement device, simplifying a calibration process of the eye movement device, and not affecting normal use of the computer by the user during calibration. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings.

[0043] Figure 1 A flowchart of the reading-based eye movement device calibration method provided by the embodiment of the present application;

[0044] Figure 2 An example diagram of a text unit provided by the embodiment of the present application;

[0045] Figure 3 An example diagram of calculation logic of gaze density provided by the embodiment of the present application;

[0046] Figure 4 An algorithm logic example diagram of a point matching process provided for an embodiment of the present application;

[0047] Figure 5 A calculation logic example diagram of gaze time estimation provided for an embodiment of the present application;

[0048] Figure 6 A calculation logic example diagram of weight of a point pair provided for an embodiment of the present application;

[0049] Figure 7 An algorithm framework example diagram of a reading-based eye movement device calibration method provided for an embodiment of the present application;

[0050] Figure 8 A structure block diagram of a reading-based eye movement device calibration system provided for an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] In the present application, the term “comprising” or “including” or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement “including a…” does not exclude the presence of another identical element in the process, method, article or device including the element.

[0053] When calibrating, an eye movement device (such as various types of eye trackers) emits infrared light to the user and collects the reflection points of the infrared light on the user's eyeball and the pupil center to form a vector, and then judges the user's gaze position. However, for different users, the correspondence between the aforementioned formed vector and the actual gaze position has a large difference, so the eye movement device needs to complete the confirmation of the correspondence through calibration before use.

[0054] Currently, eye-tracking devices typically use explicit calibration. Specifically, this involves having the user look at multiple points on the screen sequentially and acquiring the vectors as the user looks at these points. The necessary parameters for eye-tracking estimation are then calculated from these vectors, thus completing the calibration of the eye-tracking device. However, the aforementioned explicit calibration method is quite cumbersome and can interfere with the user's normal use of the computer.

[0055] The inventors discovered through research that when users read, their gaze is focused on the area where the text appears. The duration of the user's gaze also varies depending on the content of the text. By utilizing these behavioral characteristics, a calibration matrix (which is the final calibration matrix obtained later) can be calculated to move the estimated gaze position given by the uncalibrated eye-tracking device to its actual gaze position.

[0056] Based on the findings of the above research, this solution proposes a reading-based eye-tracking device calibration method, system, and related equipment. It finds the correspondence between gaze behavior and text content from the user's reading behavior while using a computer, thereby establishing an implicit relationship and completing the implicit calibration of the eye-tracking device. This simplifies the calibration process of the eye-tracking device and does not affect the user's normal use of the computer during calibration. The following detailed description of this solution is provided through various embodiments.

[0057] Before explaining this scheme, let's clarify the calibration matrices involved. The final calibration matrix mentioned later will be denoted as A, the first calibration matrix as A1, and the second calibration matrix as A2. The following explanation uses the final calibration matrix A as an example to illustrate the form and purpose of the calibration matrix.

[0058] The final calibration matrix A is a 3*3 affine matrix (specification is only for example). By multiplying the final calibration matrix A with the uncalibrated eye movement data (which includes eye movement coordinates), the calibrated eye movement data can be obtained.

[0059] The final calibration matrix A takes into account factors such as displacement, rotation, scaling, and shearing, so there are multiple parameters in the final calibration matrix A that need to be optimized. As shown in the following formula (1), the matrices of the final calibration matrix A from left to right are the displacement matrix, rotation matrix, scaling matrix, and shearing matrix, respectively.

[0060] (1);

[0061] In formula (1), trans_x and trans_y represent the displacement distance in the x and y directions, respectively. For example, trans_x=200 means moving 200 pixels in the x direction; cosθ and sinθ represent the cosine and sine values ​​of the rotation angle θ, respectively.

[0062] scale_x and scale_y represent the scaling factor in the x and y directions, respectively. For example, scale_x=1 means no scaling in the x direction, and scale_x=0.5 means scaling down to half the original size in the x direction.

[0063] shear_x and shear_y represent shearing in the x and y directions, respectively.

[0064] The concepts of trans_x, trans_y and scale_x in formula (1) are all spatial transformations represented by matrices.

[0065] The above is an explanation of the final calibration matrix A. For explanations of the first calibration matrix A1, the second calibration matrix A2, and other calibration matrices, please refer to the above explanation of the final calibration matrix A. They will not be repeated here.

[0066] See Figure 1 The diagram illustrates a flowchart of a reading-based eye-tracking device calibration method provided by an embodiment of the present invention. The eye-tracking device calibration method includes:

[0067] Step S101: Collect the uncalibrated set of eye-tracking coordinates of the user when reading specified text while wearing an eye-tracking device, and obtain the set of text unit coordinates and the set of edge unit coordinates.

[0068] In the specific implementation step S101, the user wears an eye-tracking device to collect the user's uncalibrated eye-tracking coordinate set (denoted as G) while reading the specified text, as well as the text unit coordinate set and the edge unit coordinate set. The uncalibrated eye-tracking coordinate set is also called uncalibrated eye-tracking data.

[0069] It should be noted that the uncalibrated eye-tracking coordinate set G consists of several eye-tracking coordinate points (also simply called eye-tracking coordinates, denoted as g). Each eye-tracking coordinate point g is an estimate of the user's gaze position on the screen at a certain moment when the user is looking at the screen, given by the uncalibrated eye-tracking device. Gi represents the uncalibrated eye-tracking coordinate set collected when the user is reading the i-th specified text.

[0070] The text of the specified text is displayed in several fixed text cells on the screen, such as... Figure 2 This shows several text cells fixed on the screen and the text displayed within those text cells. Figure 2 The numbers marked in the middle are in pixels. Figure 2 There are a total of 6*30=180 text units. The distance from the top left text unit to the top left corner of the screen is 360 pixels (horizontal) and 240 pixels (vertical). The size of each text unit is 60 pixels * 40 pixels.

[0071] Let t1 be the coordinates of these text units (that is, the coordinates of the center position of the text unit). All text unit coordinates t1 constitute the set of text unit coordinates (denoted as T1). Figure 2 The coordinates of the 180 text units constitute the text unit coordinate set T1, for example... Figure 2 The text cell in the upper left corner displays the content "【", and the coordinates of the text cell containing "【" are "(360+40 / 2,240+64 / 2)=(380, 272)".

[0072] Edge cells are additional "cells" added around existing text cells. They specify that when text is displayed within a text cell, any adjacent blank cells (top, bottom, left, or right) should be treated as edge cells. If the text cell is flush left (e.g., ...), then... Figure 2 The text unit in the first line (in the text) needs to be expanded outwards by another layer to generate additional edge units.

[0073] The coordinates of the edge element are also the coordinates of the center position of the edge element. The coordinates of the edge element are denoted as t2, and all edge element coordinates t2 constitute the edge element coordinate set (denoted as T2).

[0074] Step S102: Calculate the fixation density of eye coordinate points contained in the uncalibrated eye coordinate set.

[0075] In the specific implementation step S102, for each eye movement coordinate point (uncalibrated eye movement coordinate point) contained in the uncalibrated eye movement coordinate set, a fixation density (denoted as gd) is calculated for that eye movement coordinate point.

[0076] Specifically, such as Figure 3 The example diagram illustrating the calculation logic of gaze density shows that, through... Figure 3 The text describes the calculation of fixation density for uncalibrated eye movement coordinates using sections a (a) fixation sequence, b (b) fixation density calculation, and c (c) fixation density. Figure 3 The "fixation point" in the text refers to the eye movement coordinate point.

[0077] Suppose we need to calculate the fixation density of an uncalibrated eye-tracking coordinate point "g-target", we need to find the number of eye-tracking coordinate points that are always sequentially continuous with g-target and whose distance to g-target is less than a variable threshold.

[0078] like Figure 3As shown in the "a-Gaze ​​Sequence" section, the green area represents the range where the distance to the g-target is less than the variable threshold. The center of the green area is the g-target, and the other orange points are eye movement coordinates that can be included in the gaze density.

[0079] It should be noted that the temporal continuity means that for a certain eye movement coordinate point g, all eye movement coordinate points that are temporally between g and g-target are within the threshold range.

[0080] An example of temporal discontinuity is: a user initially looks at position A, then looks at position B, and finally looks at position A again. At this point, the distance between the eye movement coordinates of the initial and final eye movements when looking at position A is less than a threshold. However, since the distance between the eye movement coordinates of the intermediate eye movement when looking at position B and the eye movement coordinates of the initial eye movement when looking at position A is greater than the threshold, the two are not considered temporally continuous.

[0081] Step S103: Align the eye-tracking coordinate points with the text unit coordinates contained in the text unit coordinate set to obtain the first calibration matrix.

[0082] In the specific implementation of step S103, before formally calibrating the eye-tracking device, center point alignment is required, that is, centering the eye-tracking coordinate points (uncalibrated) with the text unit coordinates to obtain the first calibration matrix (denoted as A1).

[0083] Centering eye-tracking coordinates (uncalibrated) with text cell coordinates specifically means aligning the center of the eye-tracking data (the set of uncalibrated eye-tracking coordinates) with the center of the text in the specified text.

[0084] It should be noted that by centering the eye-tracking coordinates (uncalibrated) with the text unit coordinates, it can be ensured that the eye-tracking data and the text unit are sufficiently close.

[0085] The specific method for centering the eye-tracking coordinate points (uncalibrated) with the text unit coordinates is as follows: calculate an average point G_1 of all eye-tracking coordinate points contained in the uncalibrated eye-tracking coordinate set G, then calculate an average point T_1 of all text unit coordinates, and calculate the displacement "T_1-G_1" between the average point G_1 and the average point T_1 (which, after being converted into a matrix, becomes the first calibration matrix).

[0086] Step S104: Based on the uncalibrated eye-tracking coordinate set, text unit coordinate set, edge unit coordinate set, and gaze density, execute the point matching process, weight calculation process, and gradient descent process in sequence and repeat the process multiple times to obtain multiple second calibration matrices.

[0087] It should be noted that this solution provides a point matching process, a weight calculation process, and a gradient descent process.

[0088] In the specific implementation step S104, based on the uncalibrated eye-tracking coordinate set, text unit coordinate set, edge unit coordinate set and gaze density, the point matching process, weight calculation process and gradient descent process are executed sequentially and repeated multiple times to obtain multiple second calibration matrices.

[0089] Each execution of the point matching process, weight calculation process, and gradient descent process in sequence yields a second calibration matrix. Repeating the point matching process, weight calculation process, and gradient descent process multiple times yields multiple second calibration matrices.

[0090] The process of repeatedly executing the "point matching process, weight calculation process, and gradient descent process" is equivalent to iteratively executing the point matching process, weight calculation process, and gradient descent process.

[0091] For example, executing the point matching process, weight calculation process, and gradient descent process in sequence for each round will yield a second calibration matrix. After repeating the point matching process, weight calculation process, and gradient descent process for 100 rounds, 100 second calibration matrices (A2) can be obtained.

[0092] The point matching process, weight calculation process, and gradient descent process are explained below.

[0093] (a) Point matching process:

[0094] In some embodiments, the eye-tracking coordinate points contained in the uncalibrated eye-tracking coordinate set are matched with the text unit coordinates contained in the text unit coordinate set and the edge unit coordinates contained in the edge unit coordinate set to obtain a set of point pairs;

[0095] The point pair set contains multiple pairs of points. A pair of points contains the eye-tracking coordinate point and the text unit coordinate that are closest to each other and belong to the same row. Alternatively, a pair of points contains the eye-tracking coordinate point and the edge unit coordinate that are closest to each other.

[0096] like Figure 4 As shown in the example diagram of the point matching process, the eye-tracking coordinate points contained in the eye-tracking coordinate set are first divided into rows by clustering; then, for an eye-tracking coordinate point in a certain row, the "text unit coordinates closest to the eye-tracking coordinate point" in that row is found as the text unit coordinates that match the eye-tracking coordinate point, thus obtaining a point pair.

[0097] For edge cell coordinates, without considering clustering, the "edge cell coordinates closest to the eye-tracking coordinate point" are taken as the edge cell coordinates that match the eye-tracking coordinate point, thus obtaining point pairs.

[0098] in,Figure 4 The “gaze sequence” is a set of eye-tracking coordinates, the text center point is the text unit coordinate, and the edge center point is the edge unit coordinate.

[0099] Specifically, the point matching process requires matching each eye-tracking coordinate point g with its nearest text unit coordinate t1 and edge unit coordinate t2. For the specified texts used for calibration (i=1,2,3…), the uncalibrated eye-tracking coordinate sets collected when the user reads these specified texts are G1, G2, G3, etc. These eye-tracking coordinate sets are merged and then clustered according to the y-coordinate of the eye-tracking coordinate points.

[0100] For example, assuming the text unit in this scheme has a total of 6 lines, then the eye-tracking coordinate points will be clustered into 6 classes. After clustering, each eye-tracking coordinate point will correspond to a cluster label (j=1,2,3,4,5,6).

[0101] During the point matching process, the matching of eye-tracking coordinate point g and text unit coordinate t1 will only occur when "eye-tracking coordinate point g and text unit coordinate t1 belong to the same row". In other words, if the cluster label of an eye-tracking coordinate point is 1, then the eye-tracking coordinate point will only be matched with the text unit coordinate that is "closest to the eye-tracking coordinate point" in the first row, and will not be matched with the text unit coordinates in other rows.

[0102] Among them, the matching eye-tracking coordinate point g and the text unit coordinate t1 form a matching point pair.

[0103] For matching eye-tracking coordinates with edge cell coordinates, only distance is considered, without considering whether the eye-tracking coordinates and edge cell coordinates belong to the same row. During matching, a distance threshold is set. If the distance from the eye-tracking coordinates to the edge cell coordinates is greater than the distance threshold, a point pair will not be formed. Only when the distance is less than the distance threshold will a matching point pair be formed.

[0104] In general, after eye-tracking coordinate point g is matched with text unit coordinate t1 or edge unit coordinate t2, a pair of points (denoted as p) will be formed. This pair of points contains two coordinate information, p = (g, t), where t is the text unit coordinate t1 or the edge unit coordinate t2. All the point pairs p constitute the point pair set (denoted as P).

[0105] The above is a description of the point matching process.

[0106] (II) Weight Calculation Process:

[0107] In some embodiments, the weight of each pair of points in the set of point pairs is determined, and the specific method for determining the weight of each pair of points is as follows:

[0108] For each pair of points in the set, if the pair contains eye-tracking coordinates and text unit coordinates, and the text content corresponding to the text unit coordinates is not punctuation, calculate the gaze time estimate of the text unit coordinates contained in the pair; use the gaze density of the eye-tracking coordinates and the gaze time estimate of the text unit coordinates contained in the pair to calculate the weight of the pair.

[0109] For each pair of points in the set of point pairs, if the point pair contains eye-tracking coordinates and text cell coordinates, and the text content corresponding to the text cell coordinates contained in the point pair is punctuation, then the weight of the point pair is determined to be the first constant (e.g., -0.1), and the first constant is a negative number.

[0110] For each pair of points in the set of point pairs, if the pair contains eye-tracking coordinates and edge cell coordinates, the weight of the pair is determined to be a second constant (e.g., -0.001), and the second constant is a negative number.

[0111] Specifically, for each pair of points p in the set of point pairs P, a weight needs to be calculated for each pair of points p in order to perform subsequent gradient descent.

[0112] For a pair of points consisting of eye-tracking coordinate point g and edge unit coordinate t2, the weight of the pair is a constant.

[0113] For a pair of points consisting of eye-tracking coordinate point g and text unit coordinate t1, the weight of the pair needs to be calculated or set to a constant based on the actual situation.

[0114] When calculating the weight of a "point pair consisting of eye-tracking coordinate point g and text unit coordinate t1", it is necessary to first calculate the gaze time estimate (also known as gaze duration estimate, denoted as gdp) of the text unit coordinate. The longer the gaze time estimate of the text unit coordinate, the longer the user will gaze at the text displayed at that text unit coordinate.

[0115] like Figure 5 As shown in the example diagram illustrating the calculation logic of gaze time estimation, this scheme uses a neural network consisting of four fully connected layers to calculate gaze time. The number of neurons in each layer of this neural network is 64, 128, 256, and 512, respectively. Optimization is achieved using the Adam optimizer and the MSE (mean squared error) loss function. The training of this neural network requires 300 epochs, and the neural network has a total of 11 input features:

[0116] 1) row, representing the row number of the text cell at coordinate t1.

[0117] 2) Column: Represents the column number where the text cell coordinate t1 is located.

[0118] 3) Row length (row_length), which represents the length of the line where the text at text unit coordinate t1 is located. It should be noted that for some lines, not all text units have text. For example, in the first line of Figure 2 the row length is 6.

[0119] 4) Token index within the segmentation (token_index), which represents the sequence number of the text at text unit coordinate t1 within the segmentation. For example, the sequence number of "你" in the segmentation "你好" is 1.

[0120] 5) Token length (token_length), which represents the length of the segmentation where the text at text unit coordinate t1 is located. For example, the length of the segmentation "你好" where "你" is located is 2.

[0121] 6) Sentence index within the sentence (sentence_index). The text in each line is separated by punctuation marks, and the separated objects are sentences. For example, in Figure 2 the text in the second line can be divided into 4 sentences: "早餐", "早餐自助餐样式很丰富", "中西餐结合", "尤其是煎饼果子". The sentence index, that is, the sequence number of the text at text unit coordinate t1 within the sentence. For example, the sequence number of "自" in "早餐自助餐样式很丰富" is 3.

[0122] 7) Sentence length (sentence_length), which represents the length of the sentence where the text at text unit coordinate t1 is located. For example, the length of "早餐自助餐样式很丰富" is 10.

[0123] 8) Token embedding (token_embedding), which represents the word vector of the segmentation where the text at text unit coordinate t1 is located. This word vector has 1536 dimensions.

[0124] 9) Sentence embedding (sentence_embedding), which represents the word vector of the sentence where the text at text unit coordinate t1 is located. This word vector has 1536 dimensions.

[0125] 10) Depth for the full text (depth_for_full_text), which represents the depth of the text at text unit coordinate t1 in the constituent syntactic analysis generated for the full text.

[0126] 11) Depth for the sentence unit (depth_for_sentence_unit), which represents the depth of the text at text unit coordinate t1 in the constituent syntactic analysis generated for the sentence.

[0127] By inputting the 11 input features shown in “1)-11)” above corresponding to the text at coordinate t1 of the text unit into the neural network, the gaze time estimate gdp of the text unit coordinate predicted by the neural network can be obtained.

[0128] like Figure 6 As shown in the example diagram of the calculation logic of the point pair weight, the weight w of the point pair p consisting of "eye-tracking coordinate point" and "text unit coordinates where the text content is not punctuation" is calculated by formula (2).

[0129] (2);

[0130] In formula (2), gd is the gaze density of the eye movement coordinate point, gdp is the gaze time estimate of the text unit coordinate, and c1, c2 and c3 are preset coefficients, for example c1=5, c2=0.5, c3=3.

[0131] The weight w of the point pair p, which is composed of the "eye-tracking coordinate point" and the "text unit coordinates where the text content is not punctuation", is calculated by the above formula (2). The weight w is positive, which can ensure that the distance between the eye-tracking coordinate point g and the text unit coordinate t1 (text content is not punctuation) will be reduced during the optimization process.

[0132] For the weight w of the point pair p formed by the "eye-tracking coordinate point" and the "text unit coordinates where the text content is punctuation", the weight w of the point pair p formed by the "eye-tracking coordinate point" and the "text unit coordinates where the text content is punctuation" is directly set to the first constant (such as -0.1). This ensures that the distance between the eye-tracking coordinate point g and the text unit coordinate t1 (text content is punctuation) will be amplified during the optimization process.

[0133] For the weight w of the point pair p consisting of the "eye-tracking coordinate point" and the "edge unit coordinate", the weight w of the point pair p consisting of the "eye-tracking coordinate point" and the "edge unit coordinate" is directly set to the second constant (such as -0.001). This ensures that the distance between the eye-tracking coordinate point g and the edge unit coordinate t2 will be amplified during the optimization process.

[0134] The above is a description of the weight calculation process.

[0135] (III) Gradient Descent Process:

[0136] In some embodiments, gradient descent is performed based on the weights of eye-tracking coordinates, text unit coordinates, edge unit coordinates, and point pairs, with the optimization objective being to minimize the weighted distance from the eye-tracking coordinates to the text unit coordinates or edge unit coordinates after the eye-tracking coordinates are moved by the second calibration matrix (A2). The second calibration matrix is ​​then applied to the eye-tracking coordinates contained in the eye-tracking coordinate set G. This causes a change in the position of the eye-tracking coordinate set G, which in turn changes the matching result of the point matching process (i.e., changes the set of point pairs). This affects the calculation of the weights of the point pairs and the gradient descent. After multiple rounds of iteration, the change brought about by the second calibration matrix to the eye-tracking coordinate set G gradually decreases until it eventually converges.

[0137] In other words, this scheme uses gradient descent to solve for the second calibration matrix A2. During the gradient descent process, it is necessary to minimize the "weighted distance" of the eye-tracking coordinate point g of each point pair to the text unit coordinate t1 or the edge unit coordinate t2 after moving through the second calibration matrix. The optimization formula of gradient descent used in this scheme is formula (3).

[0138] (3);

[0139] In formula (3), A2 is the second calibration matrix, |P| is the number of "point pairs" in the point pair set, and g i Let g and t be the eye-tracking coordinate points contained in the i-th pair of points. i For the text cell coordinates t1 or edge cell coordinates t2 contained in the i-th pair of points, w i Let be the weight of the i-th pair of points.

[0140] Each time the gradient descent process is executed to obtain the second calibration matrix A2, the obtained second calibration matrix A2 is applied to the eye coordinate points contained in the eye coordinate set G. This causes the eye coordinate set G to change its position, which in turn changes the matching result of the point matching process (that is, the set of point pairs changes). This affects the calculation of the weights of the point pairs and the gradient descent. After multiple rounds of iteration, the change brought about by the second calibration matrix A2 to the eye coordinate set G will gradually decrease until it eventually converges.

[0141] The above is a description of the gradient descent process.

[0142] Step S105: Using the first calibration matrix and multiple second calibration matrices, calculate the final calibration matrix for calibrating the eye-tracking device.

[0143] In the specific implementation step S105, after repeatedly executing multiple rounds of the "point matching process, weight calculation process, and gradient descent process" to obtain multiple second calibration matrices (A2), the average value of the multiple second calibration matrices obtained from the Nth to Mth rounds of sequentially executing the point matching process, weight calculation process, and gradient descent process is calculated to obtain the third calibration matrix (denoted as A2). M is greater than N, where M can be 80 and N can be 20.

[0144] Calculate the first calibration matrix (A1) and the third calibration matrix (A2). The product of ) yields the final calibration matrix (A) used to calibrate the eye-tracking device, i.e., A = A1 * .

[0145] For example: Repeating the "point matching process, weight calculation process, and gradient descent process" for 100 rounds will yield 100 different second calibration matrices (A2). The average of the second calibration matrices (A2) obtained from the 20th (N=20) to the 80th (M=80) rounds of the "point matching process, weight calculation process, and gradient descent process" will be used to obtain the third calibration matrix. ); Final calibration matrix A = A1* .

[0146] After obtaining the final calibration matrix A, the final calibration matrix A can be used to calibrate the eye-tracking device. For example, one application scenario of this solution is to apply the final calibration matrix A to the eye-tracking coordinate points (i.e., gaze coordinates) provided by the eye-tracking device to achieve calibration.

[0147] In this embodiment of the invention, the correspondence between gaze behavior and text content is found from the user's reading behavior when using a computer, thereby establishing an implicit relationship and completing the implicit calibration of the eye-tracking device. This simplifies the calibration process of the eye-tracking device and does not affect the user's normal use of the computer during calibration.

[0148] To better understand this solution, the following will be combined with... Figure 7 The diagram illustrates an example of an algorithm framework for a reading-based eye-tracking device calibration method, with examples provided in B1 to B7. Figure 7 It includes the following parts: "a" point matching, "b" weight calculation, and "c" gradient descent and movement.

[0149] B1. Collect the uncalibrated eye movement coordinate set G of the user while reading the specified text. The uncalibrated eye movement coordinate set G consists of several eye movement coordinate points g.

[0150] For example: Figure 7 The orange point in the "point a matching" section represents the eye-tracking coordinate point g. Figure 7The transparent orange dot in the "c gradient descent and move" section represents the eye-tracking coordinate point g.

[0151] For the obtained eye-tracking coordinate set G, the second calibration matrix A2 obtained from the subsequent gradient descent process can be applied to the eye-tracking coordinate set G. This will cause a change in the position of the eye-tracking coordinate set G, which will lead to a change in the matching result of the point matching process (that is, a change in the set of point pairs). This will affect the calculation of the weights of the point pairs and the gradient descent. After multiple rounds of iteration, the change brought about by the second calibration matrix A2 to the eye-tracking coordinate set G will gradually decrease until it eventually converges.

[0152] For example: after applying the second calibration matrix A2 to the eye-tracking coordinate set G, Figure 7 The opaque orange dots in the “c gradient descent and move” section are: the eye coordinate points after the second calibration matrix A2 is applied to the eye coordinate set G (the eye coordinate points after the position transformation).

[0153] B2. Obtain the set of text cell coordinates and the set of edge cell coordinates.

[0154] It should be noted that after obtaining the final calibration matrix, the calibration results can be measured experimentally. Specifically, in the experiment, the user is required to fixate on the center point of each text unit for approximately 1 second, thus collecting eye-tracking data to be evaluated (equivalent to the set of eye-tracking coordinates to be evaluated, denoted as C). The eye-tracking coordinate points in the eye-tracking data to be evaluated are denoted as c. The final calibration matrix is ​​then applied to the eye-tracking data to be evaluated C, resulting in calibrated eye-tracking data C'. The eye-tracking coordinate points in the calibrated eye-tracking data C' are denoted as c'.

[0155] The distance between the "true coordinate position of the center point of each text unit" and the "each eye movement coordinate point c' in the calibrated eye movement data C'" is calculated to measure the calibration result of the final calibration matrix.

[0156] B3. Calculate the fixation density of eye coordinate points contained in the uncalibrated eye coordinate set.

[0157] B4. Align the eye-tracking coordinates with the text unit coordinates contained in the text unit coordinate set to obtain the first calibration matrix A1.

[0158] B5. Execution point matching process.

[0159] B6. Execute the weight calculation process.

[0160] B7. Execute the gradient descent process.

[0161] Repeating processes B5 to B7 for 100 rounds yields 100 second calibration matrices A2. The average value of the second calibration matrices A2 obtained from round 20 to round 80 is then calculated to obtain the third calibration matrix. The final calibration matrix A = A1* .

[0162] The above is an example illustrating this solution. This solution implicitly performs calibration during the user's daily use of eye-tracking devices, greatly improving the user experience of eye-tracking devices that require frequent recalibration. In practical applications, this solution can output a 3x3 final calibration matrix. Applying this final calibration matrix to the set of gaze positions provided by the uncalibrated eye-tracking device for two-dimensional spatial transformation ensures that the distance between the moved set of gaze positions and the corresponding real gaze positions is minimized.

[0163] Corresponding to the reading-based eye-tracking device calibration method provided in the above embodiments of the present invention, see also... Figure 8 The present invention also provides a structural block diagram of a reading-based eye-tracking device calibration system, which includes: a data acquisition module 100, a first calculation module 200, an alignment module 300, an execution module 400, and a second calculation module 500.

[0164] The acquisition module 100 is used to acquire the uncalibrated eye-tracking coordinate set of the user when reading specified text while wearing an eye-tracking device, as well as to obtain the text unit coordinate set and the edge unit coordinate set.

[0165] The first calculation module 200 is used to calculate the fixation density of eye coordinate points contained in the uncalibrated set of eye coordinates.

[0166] Alignment module 300 is used to center-align eye-tracking coordinate points with the text unit coordinates contained in the text unit coordinate set to obtain a first calibration matrix.

[0167] The execution module 400 is used to sequentially execute the point matching process, weight calculation process, and gradient descent process based on the uncalibrated eye movement coordinate set, text unit coordinate set, edge unit coordinate set, and gaze density, and repeat the execution multiple times to obtain multiple second calibration matrices.

[0168] The second calculation module 500 is used to calculate the final calibration matrix for calibrating the eye-tracking device using the first calibration matrix and multiple second calibration matrices.

[0169] In the specific implementation, the second calculation module 500 is specifically used to: calculate the average value of multiple second calibration matrices obtained by sequentially executing the point matching process, weight calculation process and gradient descent process from the Nth to the Mth rounds to obtain a third calibration matrix, where M is greater than N; calculate the product of the first calibration matrix and the third calibration matrix to obtain the final calibration matrix used to calibrate the eye-tracking device.

[0170] In this embodiment of the invention, the correspondence between gaze behavior and text content is found from the user's reading behavior when using a computer, thereby establishing an implicit relationship and completing the implicit calibration of the eye-tracking device. This simplifies the calibration process of the eye-tracking device and does not affect the user's normal use of the computer during calibration.

[0171] In some embodiments, the execution module 400 executes the point matching process, including:

[0172] The eye-tracking coordinates in the uncalibrated eye-tracking coordinate set are matched with the text unit coordinates in the text unit coordinate set and the edge unit coordinates in the edge unit coordinate set to obtain a set of point pairs.

[0173] The point pair set contains multiple point pairs. A point pair contains the eye-tracking coordinate point and the text cell coordinate that are closest to each other and belong to the same row, or a point pair contains the eye-tracking coordinate point and the edge cell coordinate that are closest to each other.

[0174] In other embodiments, the execution module 400 performs the weight calculation process, including: determining the weight of each pair of points in the point pair set.

[0175] Specifically, for each pair of points in the set of point pairs, if the point pair contains eye-tracking coordinates and text unit coordinates, and the text content corresponding to the text unit coordinates contained in the point pair is not punctuation, the gaze time estimate of the text unit coordinates contained in the point pair is calculated.

[0176] The weights of the point pairs are calculated by estimating the gaze density of the eye-tracking coordinates contained in the point pairs and the gaze time of the text unit coordinates.

[0177] If a point pair contains eye-tracking coordinates and text cell coordinates, and the text content corresponding to the text cell coordinates in the point pair is punctuation, then the weight of the point pair is determined as the first constant, which is a negative number.

[0178] If a point pair contains both eye-tracking coordinates and edge cell coordinates, the weight of the point pair is determined by a second constant, which is negative.

[0179] In other embodiments, the execution module 400 performs the gradient descent process, including:

[0180] Based on the weights of eye-tracking coordinates, text cell coordinates, edge cell coordinates, and point pairs, gradient descent is used to obtain the second calibration matrix by minimizing the weighted distance from the eye-tracking coordinates to the text cell coordinates or edge cell coordinates after the eye-tracking coordinates have been moved by the second calibration matrix. The second calibration matrix is ​​then applied to the eye-tracking coordinates.

[0181] Preferably, embodiments of the present invention also provide an electronic device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, specifically to implement the reading-based eye-tracking device calibration method provided in the above method embodiments.

[0182] Preferably, the present invention also provides a computer storage medium for storing a computer program, which, when executed, is specifically used to implement the reading-based eye-tracking device calibration method provided in the above-described method embodiments.

[0183] In summary, the embodiments of the present invention provide a reading-based eye-tracking device calibration method, system, and related equipment. By finding the correspondence between gaze behavior and text content from the user's reading behavior while using a computer, an implicit relationship is established, thereby completing the implicit calibration of the eye-tracking device. This simplifies the calibration process of the eye-tracking device and does not affect the user's normal use of the computer during calibration.

[0184] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0185] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0186] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for calibrating an eye-tracking device based on reading, characterized in that, The method includes: Collect the uncalibrated set of eye-tracking coordinates of the user while reading specified text wearing an eye-tracking device, and obtain the set of text unit coordinates and the set of edge unit coordinates; Calculate the fixation density of the eye coordinate points contained in the uncalibrated eye coordinate set; The eye-tracking coordinate points are aligned with the text unit coordinates contained in the text unit coordinate set to obtain the first calibration matrix; Based on the uncalibrated eye-tracking coordinate set, the text unit coordinate set, the edge unit coordinate set, and the gaze density, the point matching process, the weight calculation process, and the gradient descent process are executed sequentially and repeated multiple times to obtain multiple second calibration matrices; Using the first calibration matrix and multiple second calibration matrices, a final calibration matrix for calibrating the eye-tracking device is calculated.

2. The method according to claim 1, characterized in that, The process of executing the point matching procedure includes: The eye-tracking coordinate points contained in the uncalibrated eye-tracking coordinate set are matched with the text unit coordinates contained in the text unit coordinate set and the edge unit coordinates contained in the edge unit coordinate set to obtain a set of point pairs. The point pair set contains multiple point pairs, and each point pair contains the eye-tracking coordinate point and the text unit coordinate that are in the same row and are closest to each other, or the point pair contains the eye-tracking coordinate point and the edge unit coordinate that are closest to each other.

3. The method according to claim 2, characterized in that, The process of executing the weight calculation procedure includes: Determine the weight of each pair of points in the set of point pairs.

4. The method according to claim 3, characterized in that, Determining the weight of each pair of points in the set of point pairs includes: For each pair of points in the set of point pairs, if the point pair contains the eye-tracking coordinate point and the text unit coordinate, and the text content corresponding to the text unit coordinate contained in the point pair is not punctuation, calculate the gaze time estimate of the text unit coordinate contained in the point pair. The weight of the point pair is calculated using the gaze density of the eye movement coordinates contained in the point pair and the gaze time estimate of the text unit coordinates; If the point pair contains the eye-tracking coordinate point and the text unit coordinate, and the text content corresponding to the text unit coordinate contained in the point pair is punctuation, the weight of the point pair is determined to be a first constant, and the first constant is a negative number; If the point pair contains the eye-tracking coordinate point and the edge unit coordinate, the weight of the point pair is determined to be a second constant, where the second constant is a negative number.

5. The method according to claim 3 or 4, characterized in that, The process of performing gradient descent includes: Based on the weights of the eye-tracking coordinates, the text unit coordinates, the edge unit coordinates, and the point pairs, gradient descent is used to solve for the second calibration matrix by minimizing the weighted distance from the eye-tracking coordinates to the text unit coordinates or the edge unit coordinates after the eye-tracking coordinates have moved through the second calibration matrix. Apply the second calibration matrix to the eye-tracking coordinate point.

6. The method according to any one of claims 1-4, characterized in that, Using the first calibration matrix and multiple second calibration matrices, a final calibration matrix for calibrating the eye-tracking device is calculated, including: The average of the multiple second calibration matrices obtained from the point matching process, weight calculation process and gradient descent process executed sequentially from round N to round M is used to obtain the third calibration matrix, where M is greater than N. The product of the first calibration matrix and the third calibration matrix is ​​calculated to obtain the final calibration matrix used to calibrate the eye-tracking device.

7. A calibration system for a reading-based eye-tracking device, characterized in that, The system includes: The acquisition module is used to acquire the uncalibrated set of eye-tracking coordinates of the user when reading specified text while wearing an eye-tracking device, as well as to obtain the set of text unit coordinates and the set of edge unit coordinates; The first calculation module is used to calculate the fixation density of eye movement coordinate points contained in the uncalibrated eye movement coordinate set; An alignment module is used to center-align the eye-tracking coordinate points with the text unit coordinates contained in the text unit coordinate set to obtain a first calibration matrix; The execution module is used to sequentially execute the point matching process, the weight calculation process, and the gradient descent process based on the uncalibrated eye movement coordinate set, the text unit coordinate set, the edge unit coordinate set, and the gaze density, and repeat the execution multiple times to obtain multiple second calibration matrices. The second calculation module is used to calculate the final calibration matrix for calibrating the eye-tracking device using the first calibration matrix and a plurality of second calibration matrices.

8. The system according to claim 7, characterized in that, The execution module performs a point matching process, including: The eye-tracking coordinate points contained in the uncalibrated eye-tracking coordinate set are matched with the text unit coordinates contained in the text unit coordinate set and the edge unit coordinates contained in the edge unit coordinate set to obtain a set of point pairs. The point pair set contains multiple point pairs, and each point pair contains the eye-tracking coordinate point and the text unit coordinate that are in the same row and are closest to each other, or the point pair contains the eye-tracking coordinate point and the edge unit coordinate that are closest to each other.

9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program, specifically to implement the reading-based eye-tracking device calibration method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, Used to store a computer program, which, when executed, is specifically used to implement the reading-based eye-tracking device calibration method as described in any one of claims 1 to 6.

Images