Eye movement equipment calibration method and system based on reading and related equipment

Abstract

The invention provides a reading-based eye movement equipment calibration method and system and related equipment, and the method comprises the steps: collecting an uncalibrated eye movement coordinate set when a user wears eye movement equipment to read a specified text, and obtaining a text unit coordinate set and an edge unit coordinate set; calculating the gaze density of the eye movement coordinate points; performing center alignment on the eye movement coordinate points and text unit coordinates contained in the text unit coordinate set to obtain a first calibration matrix; sequentially executing a point matching process, a weight calculation process and a gradient descent process, and repeatedly executing for multiple rounds to obtain a plurality of second calibration matrixes; a final calibration matrix for calibrating the eye movement device is calculated using the first calibration matrix and the plurality of second calibration matrices. According to the scheme, the corresponding relation between the watching behavior and the text content is found from the reading behavior of the user using the computer, so that implicit calibration of the eye movement equipment is completed, the calibration process of the eye movement equipment is simplified, and normal use of the user on the computer is not affected during calibration.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for calibrating an eye movement device based on reading, and related equipment. Background Art

[0002] Currently, eye movement devices (various types of eye trackers) are usually calibrated using an explicit calibration method. Specifically, the user is asked to successively gaze at multiple points on the screen and the vectors of the user's gaze at these points are obtained. The necessary parameters in the eye movement estimation process are solved through the obtained vectors, thereby completing the calibration of the eye movement device.

[0003] However, the calibration process of the above-mentioned explicit calibration method is relatively cumbersome and will interfere with the user's normal use of the computer. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a reading-based eye movement device calibration method, system and related equipment to simplify the eye movement device calibration process and avoid interfering with the user's normal use of the computer during calibration.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a method for calibrating an eye movement device based on reading, the method comprising:

[0007] Collecting an uncalibrated eye movement coordinate set when a user wears an eye movement device to read a specified text, and obtaining a text unit coordinate set and an edge unit coordinate set;

[0008] Calculating the gaze density of the eye movement coordinate points included in the uncalibrated eye movement coordinate set;

[0009] Center-aligning the eye movement coordinate point with the text unit coordinates included in the text unit coordinate set to obtain a first calibration matrix;

[0010] According to the uncalibrated eye movement coordinate set, the text unit coordinate set, the edge unit coordinate set and the gaze density, sequentially execute a point matching process, a weight calculation process and a gradient descent process and repeat the process for multiple rounds to obtain multiple second calibration matrices;

[0011] The first calibration matrix and a plurality of the second calibration matrices are used to calculate a final calibration matrix for calibrating the eye movement device.

[0012] Preferably, the process of executing the point matching process includes:

[0013] Matching the eye movement coordinate points included in the uncalibrated eye movement coordinate set with the text unit coordinates included in the text unit coordinate set and the edge unit coordinates included in the edge unit coordinate set, respectively, to obtain a point pair set;

[0014] The point pair set includes multiple groups of point pairs, and the point pairs include the eye movement coordinate points and the text unit coordinates that belong to the same row and are closest to each other, or the point pairs include the eye movement coordinate points and the edge unit coordinates that are closest to each other.

[0015] Preferably, the process of executing the weight calculation process includes:

[0016] The weight of each group of the point pairs in the point pair set is determined.

[0017] Preferably, determining the weight of each group of the point pairs in the point pair set includes:

[0018] For each group of the point pairs in the point pair set, if the point pairs include the eye movement coordinate points and the text unit coordinates, and the text content corresponding to the text unit coordinates included in the point pairs is not a punctuation, calculating the fixation time estimation of the text unit coordinates included in the point pairs;

[0019] The weight of the point pair is calculated by using the gaze density of the eye movement coordinate point contained in the point pair and the gaze time estimation of the text unit coordinate;

[0020] If the point pair includes the eye movement coordinate point and the text unit coordinate, and the text content corresponding to the text unit coordinate included in the point pair is a punctuation mark, determining the weight of the point pair to be a first constant, and the first constant is a negative number;

[0021] If the point pair includes the eye movement coordinate point and the edge unit coordinate, the weight of the point pair is determined to be a second constant, and the second constant is a negative number.

[0022] Preferably, the process of performing the gradient descent process includes:

[0023] According to the eye movement coordinate point, the text unit coordinate, the edge unit coordinate and the weight of the point pair, a gradient descent solution is performed with the minimum weighted distance from the eye movement coordinate point to the text unit coordinate or the edge unit coordinate after being moved through the second calibration matrix as the optimization goal, to obtain the second calibration matrix;

[0024] Applying the second calibration matrix to the eye movement coordinate points.

[0025] Preferably, using the first calibration matrix and a plurality of the second calibration matrices to calculate a final calibration matrix for calibrating the eye movement device comprises:

[0026] averaging a plurality of the second calibration matrices obtained by sequentially executing the point matching process, the weight calculation process, and the gradient descent process from the Nth round to the Mth round to obtain a third calibration matrix, where M is greater than N;

[0027] The product of the first calibration matrix and the third calibration matrix is ​​calculated to obtain a final calibration matrix for calibrating the eye movement device.

[0028] A second aspect of an embodiment of the present invention discloses a reading-based eye movement device calibration system, the system comprising:

[0029] A collection module, used to collect an uncalibrated eye movement coordinate set when a user wears an eye movement device to read a specified text, and to obtain a text unit coordinate set and an edge unit coordinate set;

[0030] A first calculation module, configured to calculate the gaze density of the eye movement coordinate points included in the uncalibrated eye movement coordinate set;

[0031] an alignment module, used for center-aligning the eye movement coordinate point with the text unit coordinates included in the text unit coordinate set to obtain a first calibration matrix;

[0032] An execution module, configured to sequentially execute a point matching process, a weight calculation process, and a gradient descent process according to the uncalibrated eye movement coordinate set, the text unit coordinate set, the edge unit coordinate set, and the gaze density, and repeat the execution for multiple rounds to obtain multiple second calibration matrices;

[0033] The second calculation module is used to calculate a final calibration matrix for calibrating the eye movement device by using the first calibration matrix and a plurality of the second calibration matrices.

[0034] Preferably, the process of the execution module executing the point matching process includes:

[0035] Matching the eye movement coordinate points included in the uncalibrated eye movement coordinate set with the text unit coordinates included in the text unit coordinate set and the edge unit coordinates included in the edge unit coordinate set, respectively, to obtain a point pair set;

[0036] The point pair set includes multiple groups of point pairs, and the point pairs include the eye movement coordinate points and the text unit coordinates that belong to the same row and are closest to each other, or the point pairs include the eye movement coordinate points and the edge unit coordinates that are closest to each other.

[0037] A third aspect of an embodiment of the present invention discloses an electronic device, including a memory and a processor;

[0038] The memory is used to store computer programs;

[0039] The processor is used to execute the computer program, specifically to implement the reading-based eye movement device calibration method disclosed in the first aspect of the embodiment of the present invention.

[0040] A fourth aspect of an embodiment of the present invention discloses a computer storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the reading-based eye movement device calibration method disclosed in the first aspect of the embodiment of the present invention.

[0041] Based on the above-mentioned embodiment of the present invention, a method, system and related equipment for eye movement device calibration based on reading are provided. The method is as follows: collecting an uncalibrated eye movement coordinate set of a user when wearing an eye movement device to read a specified text, and obtaining a text unit coordinate set and an edge unit coordinate set; calculating the gaze density of the eye movement coordinate points contained in the uncalibrated eye movement coordinate set; center-aligning the eye movement coordinate points with the text unit coordinates contained in the text unit coordinate set to obtain a first calibration matrix; sequentially executing a point matching process, a weight calculation process and a gradient descent process and repeating them for multiple rounds to obtain multiple second calibration matrices; using the first calibration matrix and multiple second calibration matrices, the final calibration matrix for calibrating the eye movement device is calculated. This solution finds the corresponding relationship between the gaze behavior and the text content from the user's reading behavior using the computer, thereby completing the establishment of an implicit relationship, and then completing the implicit calibration of the eye movement device, streamlining the calibration process of the eye movement device, and not affecting the user's normal use of the computer during calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flowchart of a method for calibrating an eye movement device based on reading provided by an embodiment of the present invention;

[0044] Figure 2 An example diagram of a text unit provided by an embodiment of the present invention;

[0045] Figure 3 An example diagram of the calculation logic of the gaze density provided by an embodiment of the present invention;

[0046] Figure 4 An example diagram of the algorithm logic of the point matching process provided by an embodiment of the present invention;

[0047] Figure 5 An example diagram of calculation logic for gaze time estimation provided by an embodiment of the present invention;

[0048] Figure 6 An example diagram of the calculation logic of the weight of a point pair provided in an embodiment of the present invention;

[0049] Figure 7 An example diagram of an algorithm framework of a reading-based eye movement device calibration method provided by an embodiment of the present invention;

[0050] Figure 8 A structural block diagram of a reading-based eye movement device calibration system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0053] When calibrating an eye-tracking device (such as various types of eye trackers), it emits infrared light to the user and collects the vector formed by the reflection point of the infrared light on the user's eyeball and the center of the pupil to determine the user's gaze position. However, for different users, the correspondence between the aforementioned vector and the actual gaze position varies greatly, so the eye-tracking device needs to be calibrated before use to complete the confirmation of this correspondence.

[0054] At present, eye movement devices are usually calibrated by explicit calibration. Specifically, the user is asked to successively look at multiple points on the screen and obtain the vectors when the user looks at these points. The necessary parameters in the eye movement estimation process are solved by the obtained vectors, thereby completing the calibration of the eye movement device. However, the calibration process of the explicit calibration method mentioned above is relatively cumbersome and will interfere with the user's normal use of the computer.

[0055] The inventors have discovered through research that when a user is reading, the user's line of sight will be focused on the area where the text appears. Depending on the content of the text, the user's gaze duration will also vary. By using these behavioral characteristics, it is possible to calculate a calibration matrix required to move the estimate of the gaze position given by an uncalibrated eye movement device to its actual gaze position (that is, the final calibration matrix obtained later in the text).

[0056] Based on the above research findings, this solution proposes a reading-based eye movement device calibration method, system and related equipment, which finds the corresponding relationship between gaze behavior and text content from the user's reading behavior when using a computer, thereby completing the establishment of an implicit relationship, and then completing the implicit calibration of the eye movement device, streamlining the calibration process of the eye movement device, and not affecting the user's normal use of the computer during calibration. The following is a detailed description of this solution through various embodiments.

[0057] Before explaining this solution, the calibration matrix involved in this solution is explained first, wherein the final calibration matrix mentioned later is recorded as A, the first calibration matrix mentioned later is recorded as A1, and the second calibration matrix mentioned later is recorded as A2. The form and use of the calibration matrix are explained below using the final calibration matrix A as an example.

[0058] The final calibration matrix A is a 3*3 (specifications are for example only) affine matrix. After multiplying the final calibration matrix A with the uncalibrated eye movement data (including the eye movement coordinate points), the calibrated eye movement data can be obtained.

[0059] The final calibration matrix A takes into account factors such as displacement, rotation, scaling and shear, so there are multiple parameters that need to be optimized in the final calibration matrix A. As shown in the following formula (1), the matrices of the final calibration matrix A from left to right are the displacement matrix, rotation matrix, scaling matrix and shear matrix.

[0060] (1);

[0061] In formula (1), trans_x and trans_y represent the displacement distance in the x direction and y direction respectively. For example, trans_x=200 represents a movement of 200 pixels in the x direction. cosθ and sinθ represent the cosine and sine of the rotation angle θ respectively.

[0062] scale_x and scale_y represent the scaling factors in the x direction and y direction respectively. For example, scale_x=1 means no scaling in the x direction, and scale_x=0.5 means scaling down to half of the original in the x direction.

[0063] shear_x and shear_y represent the shear in the x-direction and y-direction respectively.

[0064] The concepts of trans_x, trans_y, and scale_x in formula (1) are all concepts that represent spatial transformation through matrices.

[0065] The above is an explanation of the final calibration matrix A. For the explanation of the first calibration matrix A1 and the second calibration matrix A2 and other calibration matrices, please refer to the above explanation of the final calibration matrix A, which will not be repeated here.

[0066] See also Figure 1 , shows a flow chart of a reading-based eye movement device calibration method provided by an embodiment of the present invention, the eye movement device calibration method comprising:

[0067] Step S101: collecting an uncalibrated eye movement coordinate set when a user wears an eye movement device and reads a specified text, and obtaining a text unit coordinate set and an edge unit coordinate set.

[0068] In the specific implementation of step S101, the user wears an eye movement device to collect an uncalibrated eye movement coordinate set (denoted as G) of the user when reading a specified text, as well as a text unit coordinate set and an edge unit coordinate set. The uncalibrated eye movement coordinate set is also called uncalibrated eye movement data.

[0069] It should be noted that the uncalibrated eye movement coordinate set G consists of a number of eye movement coordinate points (also referred to as eye movement coordinates, denoted as g), each of which is an estimate of the gaze position on the screen given by the uncalibrated eye movement device at a certain moment when the user looks at the screen. Gi represents the uncalibrated eye movement coordinate set collected when the user reads the i-th specified text.

[0070] The characters of the specified text are displayed in several fixed text units on the screen, such as Figure 2 It shows several text units fixed on the screen and the text displayed in the text units. Figure 2 The numbers marked in the figure are in pixels. Figure 2 There are a total of 6*30=180 text units. The distance from the upper left corner of the text unit to the upper left corner of the screen is 360 pixels (horizontally) and 240 pixels (vertically). The size of each text unit is 60 pixels*40 pixels.

[0071] The text unit coordinates of these text units (that is, the center position coordinates of the text units) are recorded as t1, and all text unit coordinates t1 constitute a text unit coordinate set (recorded as T1), that is, Figure 2 The text unit coordinates of the 180 text units in constitute the text unit coordinate set T1, for example Figure 2 The content displayed in the text unit in the upper left corner is "【", and the text unit coordinates of "【" are "(360+40 / 2,240+64 / 2)=(380, 272)".

[0072] Margin cells are an extra layer of "cells" added to the periphery of existing text cells. When the specified text is displayed in a text cell, as long as there are adjacent blank cells above, below, left, and right, the blank cells need to be turned into margin cells. Figure 2 ), then one more layer needs to be expanded outward to generate additional edge units.

[0073] The edge cell coordinates of the edge cell are also the center position coordinates of the edge cell. The edge cell coordinates are labeled t2, and all edge cell coordinates t2 constitute an edge cell coordinate set (labeled as T2).

[0074] Step S102: Calculate the gaze density of the eye movement coordinate points included in the uncalibrated eye movement coordinate set.

[0075] In the specific implementation of step S102 , for each eye movement coordinate point (uncalibrated eye movement coordinate point) included in the uncalibrated eye movement coordinate set, a gaze density (denoted as gd) is calculated for the eye movement coordinate point.

[0076] Specifically, if Figure 3 The calculation logic example diagram of the gaze density is shown in FIG. Figure 3 The gaze density of the uncalibrated eye movement coordinate points is calculated by using the "a. gaze point sequence", "b. gaze point density calculation" and "c. gaze point density" sections in the . Figure 3 The "fixation point" in the graph is the eye movement coordinate point.

[0077] Assume that we need to calculate the gaze density of an uncalibrated eye movement coordinate point called “g-target”. We need to find the number of eye movement coordinate points that are always continuous with g-target in time sequence and whose distance to g-target is less than a variable threshold.

[0078] like Figure 3As shown in the "a fixation point sequence" section of the image, the green area indicates the range where the distance to the g-target is less than the variable threshold. The center of the green area is the g-target, and the other orange points are eye movement coordinate points that can be counted into the fixation density.

[0079] It should be noted that the temporal continuity always indicates that for a certain eye movement coordinate point g, the eye movement coordinate points between g and g-target in the temporal sequence are all within the threshold range.

[0080] An example of temporal discontinuity is: the user initially looks at position A, then looks at position B, and finally looks at position A again. At this time, the distance between the eye movement coordinate points at the beginning and the end of looking at position A is less than the threshold, but because the distance between the eye movement coordinate point when looking at position B in the middle and the eye movement coordinate point when looking at position A at the beginning is greater than the threshold, the two are not considered to be temporally continuous.

[0081] Step S103: center-aligning the eye movement coordinate point with the text unit coordinates included in the text unit coordinate set to obtain a first calibration matrix.

[0082] In the specific implementation of step S103 , before formally calibrating the eye movement device, it is necessary to first perform center point alignment, that is, to align the eye movement coordinate point (uncalibrated) with the text unit coordinate, so as to obtain a first calibration matrix (denoted as A1).

[0083] Aligning the center of the eye movement coordinate point (uncalibrated) with the text unit coordinate specifically means: aligning the center of the eye movement data (uncalibrated eye movement coordinate set) with the center of the text of the specified text.

[0084] It should be noted that by aligning the center of the eye movement coordinate point (uncalibrated) with the text unit coordinate, it can be ensured that the eye movement data and the text unit are close enough.

[0085] The specific method of center-aligning the eye movement coordinate points (uncalibrated) with the text unit coordinates is as follows: calculate an average point G_1 of all eye movement coordinate points contained in the uncalibrated eye movement coordinate set G, then calculate an average point T_1 of all text unit coordinates, and calculate the displacement "T_1-G_1" between the average point G_1 and the average point T_1 (converting it into a matrix is ​​the first calibration matrix).

[0086] Step S104: according to the uncalibrated eye movement coordinate set, text unit coordinate set, edge unit coordinate set and gaze density, execute the point matching process, weight calculation process and gradient descent process in sequence and repeat them for multiple rounds to obtain multiple second calibration matrices.

[0087] It should be noted that this solution provides a point matching process, a weight calculation process, and a gradient descent process.

[0088] In the process of implementing step S104, the point matching process, the weight calculation process and the gradient descent process are executed in sequence according to the uncalibrated eye movement coordinate set, the text unit coordinate set, the edge unit coordinate set and the gaze density, and repeated for multiple rounds to obtain multiple second calibration matrices.

[0089] Among them, each round of point matching process, weight calculation process and gradient descent process is executed in sequence to obtain a second calibration matrix, and multiple rounds of "point matching process, weight calculation process and gradient descent process" are repeated to obtain multiple second calibration matrices.

[0090] Among them, repeatedly executing multiple rounds of "point matching process, weight calculation process and gradient descent process" is equivalent to cyclically iterating the execution of the point matching process, weight calculation process and gradient descent process.

[0091] For example, a second calibration matrix can be obtained by sequentially executing a round of point matching process, weight calculation process and gradient descent process. After repeating 100 rounds of "point matching process, weight calculation process and gradient descent process", 100 second calibration matrices (A2) can be obtained.

[0092] The point matching process, weight calculation process and gradient descent process are explained below.

[0093] 1. Point matching process:

[0094] In some embodiments, the eye movement coordinate points included in the uncalibrated eye movement coordinate set are matched with the text unit coordinates included in the text unit coordinate set and the edge unit coordinates included in the edge unit coordinate set to obtain a point pair set;

[0095] The point pair set includes multiple groups of point pairs, one group of point pairs includes eye movement coordinate points and text unit coordinates that belong to the same row and are closest to each other, or one group of point pairs includes eye movement coordinate points and edge unit coordinates that are closest to each other.

[0096] like Figure 4 As can be seen from the algorithm logic example diagram of the point matching process, the eye movement coordinate points contained in the eye movement coordinate set are first divided into rows by clustering; then for the eye movement coordinate points in a certain row, the "text unit coordinates closest to the eye movement coordinate point" are found in the row as the text unit coordinates matching the eye movement coordinate point, thereby obtaining a point pair.

[0097] For edge unit coordinates, clustering is not considered, and “edge unit coordinates closest to the eye movement coordinate point” are used as edge unit coordinates matching the eye movement coordinate point, thereby obtaining a point pair.

[0098] in, Figure 4 The "fixation point sequence" in is a set of eye movement coordinates, the text center point is the text unit coordinate, and the edge center point is the edge unit coordinate.

[0099] Specifically, the point matching process needs to match each eye movement coordinate point g with its nearest text unit coordinate t1 and edge unit coordinate t2. For the specified texts (i=1,2,3…) required for calibration, the uncalibrated eye movement coordinate sets collected when the user reads these specified texts are G1, G2, G3, etc. These eye movement coordinate sets are merged and then clustered according to the y coordinates of the eye movement coordinate points.

[0100] For example, if the text unit of this scheme has a total of 6 lines, then the eye movement coordinate points are clustered into 6 clusters. After clustering, each eye movement coordinate point will correspond to a cluster label (j=1,2,3,4,5,6).

[0101] When executing the point matching process, the matching of the eye movement coordinate point g and the text unit coordinate t1 will only occur when "the eye movement coordinate point g and the text unit coordinate t1 belong to the same row". That is to say, if the cluster label of an eye movement coordinate point is 1, then the eye movement coordinate point will only match the text unit coordinates "closest to the eye movement coordinate point" in the first row, and will not match the text unit coordinates in other rows.

[0102] The matching eye movement coordinate point g and the text unit coordinate t1 form a matching point pair.

[0103] For the matching of eye movement coordinate points and edge unit coordinates, only the distance is considered, regardless of whether the eye movement coordinate points and edge unit coordinates belong to the same row. When matching, a distance threshold is set. If the distance from the eye movement coordinate point to the edge unit coordinate is greater than the distance threshold, no point pair will be formed. Only when the distance is less than the distance threshold will a matching point pair be formed.

[0104] In general, when the eye movement coordinate point g is matched with the text unit coordinate t1 or the edge unit coordinate t2, a set of point pairs (denoted as p) will be formed. The point pair contains two coordinate information, p=(g, t), where t is the text unit coordinate t1 or the edge unit coordinate t2. All point pairs p constitute a point pair set (denoted as P).

[0105] The above is the description of the point matching process.

[0106] (II) Weight calculation process:

[0107] In some embodiments, the weight of each group of point pairs in the point pair set is determined, and the specific method of determining the weight of each group of point pairs is:

[0108] For each point pair in the point pair set, if the point pair contains eye movement coordinate points and text unit coordinates, and the text content corresponding to the text unit coordinates contained in the point pair is not punctuation, calculate the gaze time estimate of the text unit coordinates contained in the point pair; use the gaze density of the eye movement coordinate points contained in the point pair and the gaze time estimate of the text unit coordinates to calculate the weight of the point pair.

[0109] For each point pair in the point pair set, if the point pair contains eye movement coordinate points and text unit coordinates, and the text content corresponding to the text unit coordinates contained in the point pair is punctuation, the weight of the point pair is determined to be a first constant (such as -0.1), and the first constant is a negative number.

[0110] For each point pair in the point pair set, if the point pair includes an eye movement coordinate point and an edge unit coordinate, the weight of the point pair is determined to be a second constant (eg -0.001), and the second constant is a negative number.

[0111] Specifically, for each set of point pairs p in the point pair set P, a weight needs to be calculated for each set of point pairs p for subsequent gradient descent.

[0112] For a point pair consisting of the eye movement coordinate point g and the edge unit coordinate t2, the weight of the point pair is a constant.

[0113] For a point pair formed by the eye movement coordinate point g and the text unit coordinate t1, the weight of the point pair needs to be calculated or set to a constant according to actual conditions.

[0114] When calculating the weight of the "point pair formed by the eye movement coordinate point g and the text unit coordinate t1", it is necessary to first calculate the gaze time estimate of the text unit coordinate (also called the gaze duration estimate, denoted as GDP). The longer the gaze time estimate of the text unit coordinate, the longer the user will gaze at the text displayed at the text unit coordinate.

[0115] like Figure 5 As shown in the example diagram of the calculation logic of the gaze time estimation, this scheme uses a neural network consisting of 4 fully connected layers to realize the calculation of the gaze time estimation. The number of neurons in each layer of the neural network is 64, 128, 256 and 512 respectively. The Adam optimizer and MSE (mean square error) loss function are used to achieve optimization. The training of the neural network requires 300 rounds. The neural network has a total of 11 input features as follows:

[0116] 1) Row: represents the row number where the text unit coordinate t1 is located.

[0117] 2) Column: represents the column number where the text unit coordinate t1 is located.

[0118] 3) Row length (row_length), representing the length of the line where the text at text unit coordinate t1 is located. It should be noted that for some lines, not all text units have text. For example, in the first line of Figure 2 the row length is 6.

[0119] 4) Token index within the token (token_index), representing the serial number of the text at text unit coordinate t1 within the token it belongs to. For example, the serial number of "你" in the token "你好" is 1.

[0120] 5) Token length (token_length), representing the length of the token where the text at text unit coordinate t1 is located. For example, the length of the token "你好" where "你" is located is 2.

[0121] 6) Sentence index within the sentence (sentence_index). By separating the text in each line with punctuation marks, the separated objects are sentences. For example, in Figure 2 the text in the second line can be divided into 4 sentences: "早餐", "早餐自助餐样式很丰富", "中西餐结合", "尤其是煎饼果子". The sentence index, that is, the serial number of the text at text unit coordinate t1 within the sentence it belongs to. For example, the serial number of "自" in "早餐自助餐样式很丰富" is 3.

[0122] 7) Sentence length (sentence_length), indicating the length of the sentence where the text at text unit coordinate t1 is located. For example, the length of "早餐自助餐样式很丰富" is 10.

[0123] 8) Token embedding (token_embedding), representing the word vector of the token where the text at text unit coordinate t1 is located. This word vector has 1536 dimensions.

[0124] 9) Sentence embedding (sentence_embedding), representing the word vector of the sentence where the text at text unit coordinate t1 is located. This word vector has 1536 dimensions.

[0125] 10) Depth for the full text (depth_for_full_text), representing the depth of the text at text unit coordinate t1 in the constituent syntactic analysis generated for the full text.

[0126] 11) Depth for the sentence unit (depth_for_sentence_unit), representing the depth of the text at text unit coordinate t1 in the constituent syntactic analysis generated for the sentence.

[0127] By inputting the 11 input features shown in the above “1)-11)” corresponding to the text at the text unit coordinate t1 into the neural network, the fixation time estimation GDP of the text unit coordinate predicted by the neural network can be obtained.

[0128] like Figure 6 As shown in the example diagram of the calculation logic of the weight of the point pair, it can be seen that the weight w of the point pair p composed of the "eye movement coordinate point" and the "text unit coordinate of the text content that is not a punctuation mark" is calculated by formula (2).

[0129] (2);

[0130] In formula (2), gd is the gaze density of the eye movement coordinate point, gdp is the estimated gaze time of the text unit coordinate, c1, c2 and c3 are preset coefficients, for example, c1=5, c2=0.5, c3=3.

[0131] The weight w of the point pair p formed by the "eye movement coordinate point" and the "text unit coordinate whose text content is not punctuation" calculated by the above formula (2) is positive, which can ensure that the distance between the eye movement coordinate point g and the text unit coordinate t1 (text content is not punctuation) will be reduced during the optimization process.

[0132] For the weight w of the point pair p formed by the "eye movement coordinate point" and the "text unit coordinates with punctuation as the text content", the weight w of the point pair p formed by the "eye movement coordinate point" and the "text unit coordinates with punctuation as the text content" is directly set to the first constant (such as -0.1). This can ensure that the distance between the eye movement coordinate point g and the text unit coordinate t1 (the text content is punctuation) will be enlarged during the optimization process.

[0133] For the weight w of the point pair p composed of the "eye movement coordinate point" and the "edge unit coordinate", the weight w of the point pair p composed of the "eye movement coordinate point" and the "edge unit coordinate" is directly set to the second constant (such as -0.001), which can ensure that the distance between the eye movement coordinate point g and the edge unit coordinate t2 will be enlarged during the optimization process.

[0134] The above is an explanation of the weight calculation process.

[0135] (III) Gradient descent process:

[0136] In some embodiments, according to the weights of eye movement coordinate points, text unit coordinates, edge unit coordinates and point pairs, a gradient descent solution is performed with the minimum weighted distance from the eye movement coordinate points to the text unit coordinates or the edge unit coordinates after moving through the second calibration matrix (A2) as the optimization goal, to obtain a second calibration matrix, and the second calibration matrix is ​​applied to the eye movement coordinate points included in the eye movement coordinate set G, so that the eye movement coordinate set G will undergo a position transformation, thereby causing the matching result of the point matching process to change (that is, the point pair set changes), thereby affecting the calculation of the point pair weights and affecting the gradient descent. After multiple rounds of iterations, the changes brought about by the second calibration matrix to the eye movement coordinate set G will gradually become smaller until they finally converge.

[0137] That is to say, this solution uses gradient descent to solve the required second calibration matrix A2. During the gradient descent process, it is necessary to minimize the "weighted distance" from the eye movement coordinate point g of each point pair to the text unit coordinate t1 or the edge unit coordinate t2 after moving through the second calibration matrix. The optimization formula of the gradient descent used in this solution is formula (3).

[0138] (3);

[0139] In formula (3), A2 is the second calibration matrix, |P| is the number of “point pairs” in the point pair set, and g i is the eye movement coordinate point g, t contained in the i-th group of points i is the text unit coordinate t1 or edge unit coordinate t2 contained in the i-th group of points, w i is the weight of the i-th group of point pairs.

[0140] Each time the gradient descent process is executed to obtain the second calibration matrix A2, the second calibration matrix A2 obtained this time is applied to the eye movement coordinate points contained in the eye movement coordinate set G. In this way, the eye movement coordinate set G will undergo a position transformation, which will cause the matching result of the point matching process to change (that is, the point pair set changes), thereby affecting the calculation of the point pair weights and the gradient descent. After multiple rounds of iterations, the changes brought about by the second calibration matrix A2 to the eye movement coordinate set G will gradually become smaller until they finally converge.

[0141] The above is an explanation of the gradient descent process.

[0142] Step S105: using the first calibration matrix and the plurality of second calibration matrices, a final calibration matrix for calibrating the eye movement device is calculated.

[0143] In the specific implementation of step S105, after repeatedly executing multiple rounds of "point matching process, weight calculation process and gradient descent process" to obtain multiple second calibration matrices (A2), the average value of the multiple second calibration matrices obtained by sequentially executing the point matching process, weight calculation process and gradient descent process from the Nth round to the Mth round is calculated to obtain a third calibration matrix (denoted as ), M is greater than N, M can be 80 and N can be 20.

[0144] Calculate the first calibration matrix (A1) and the third calibration matrix ( ) to obtain the final calibration matrix (A) used to calibrate the eye tracking device, that is, A=A1* .

[0145] For example, repeatedly executing the “point matching process, weight calculation process, and gradient descent process” will result in 100 different second calibration matrices (A2) after 100 rounds of repetition. The second calibration matrices (A2) obtained by executing the “point matching process, weight calculation process, and gradient descent process” from the 20th (N=20) round to the 80th (M=80) round will be averaged to obtain the third calibration matrix ( ); Final calibration matrix A=A1* .

[0146] After obtaining the final calibration matrix A, the final calibration matrix A can be used to calibrate the eye movement device. For example, one of the application scenarios of this solution is to apply the final calibration matrix A to the eye movement coordinate points (that is, gaze coordinates) given by the eye movement device to achieve calibration.

[0147] In an embodiment of the present invention, the corresponding relationship between gaze behavior and text content is found from the user's reading behavior when using a computer, thereby completing the establishment of an implicit relationship, and then completing the implicit calibration of the eye movement device, streamlining the calibration process of the eye movement device, and not affecting the user's normal use of the computer during calibration.

[0148] To better understand this solution, the following Figure 7 An example diagram of an algorithm framework of a reading-based eye movement device calibration method is shown, and is illustrated by B1 to B7, wherein: Figure 7 It includes "a point matching", "b weight calculation" and "c gradient descent and movement".

[0149] B1. Collecting an uncalibrated eye movement coordinate set G of the user when reading a specified text. The uncalibrated eye movement coordinate set G is composed of a plurality of eye movement coordinate points g.

[0150] For example: Figure 7 The orange point in the "a point matching" section is the eye movement coordinate point g. Figure 7The transparent orange dot in the “c Gradient Descent and Move” section is the eye movement coordinate point g.

[0151] For the obtained eye movement coordinate set G, the second calibration matrix A2 obtained by the subsequent execution of the gradient descent process can be applied to the eye movement coordinate set G, so that the eye movement coordinate set G will undergo a position transformation, thereby causing the matching result of the point matching process to change (that is, the point pair set changes), thereby affecting the calculation of the point pair weights and affecting the gradient descent. After multiple rounds of iterations, the changes brought about by the second calibration matrix A2 to the eye movement coordinate set G will gradually become smaller until they finally converge.

[0152] For example, after applying the second calibration matrix A2 to the eye movement coordinate set G, Figure 7 The opaque orange dots in the “c gradient descent and move” part are the eye movement coordinate points after the second calibration matrix A2 is applied to the eye movement coordinate set G (the eye movement coordinate points after the position transformation occurs).

[0153] B2. Obtain a text unit coordinate set and an edge unit coordinate set.

[0154] It should be noted that after the final calibration matrix is ​​obtained, in order to measure the calibration result of the final calibration matrix, the calibration result can be measured by experiment. The specific measurement method is: in the experiment, the user is required to look at the center point of each text unit and the looking time is about 1 second, so that the eye movement data to be evaluated (equivalent to the eye movement coordinate set to be evaluated, denoted as C) is collected, and the eye movement coordinate point in the eye movement data to be evaluated is denoted as c. The final calibration matrix is ​​applied to the eye movement data to be evaluated C to obtain the calibrated eye movement data C', and the eye movement coordinate point in the calibrated eye movement data C' is denoted as c'.

[0155] The distance between the “real coordinate position of the center point of each text unit” and the “each eye movement coordinate point c' in the calibrated eye movement data C'” is calculated to measure the calibration result of the final calibration matrix.

[0156] B3. Calculate the gaze density of the eye movement coordinate points contained in the uncalibrated eye movement coordinate set.

[0157] B4. Align the eye movement coordinate point with the text unit coordinates included in the text unit coordinate set to obtain a first calibration matrix A1.

[0158] B5. Execute point matching process.

[0159] B6. Execute the weight calculation process.

[0160] B7. Execute the gradient descent process.

[0161] Repeat the process B5 to B7, and after 100 rounds, 100 second calibration matrices A2 can be obtained. The second calibration matrices A2 obtained from the 20th to the 80th rounds are averaged to obtain the third calibration matrix , the final calibration matrix A=A1* .

[0162] The above is an example of this solution. This solution implicitly completes the calibration in the process of daily use of eye tracking devices by users, greatly improving the user experience of eye tracking devices that require frequent repeated calibration. In practical applications, this solution can output a 3*3 final calibration matrix, and apply the final calibration matrix to the gaze position set given by the uncalibrated eye tracking device for two-dimensional spatial transformation, which can ensure that the distance between the gaze position set after the movement and the corresponding real gaze position is the shortest.

[0163] Corresponding to the eye movement device calibration method based on reading provided by the above embodiment of the present invention, see Figure 8 , an embodiment of the present invention also provides a structural block diagram of a reading-based eye movement device calibration system, the eye movement device calibration system comprising: an acquisition module 100, a first calculation module 200, an alignment module 300, an execution module 400 and a second calculation module 500;

[0164] The acquisition module 100 is used to acquire an uncalibrated eye movement coordinate set when a user wears an eye movement device and reads a specified text, and to obtain a text unit coordinate set and an edge unit coordinate set.

[0165] The first calculation module 200 is used to calculate the gaze density of the eye movement coordinate points included in the uncalibrated eye movement coordinate set.

[0166] The alignment module 300 is used to perform center alignment between the eye movement coordinate point and the text unit coordinates included in the text unit coordinate set to obtain a first calibration matrix.

[0167] The execution module 400 is used to execute the point matching process, the weight calculation process and the gradient descent process in sequence and repeat multiple rounds according to the uncalibrated eye movement coordinate set, the text unit coordinate set, the edge unit coordinate set and the gaze density to obtain multiple second calibration matrices.

[0168] The second calculation module 500 is used to calculate a final calibration matrix for calibrating the eye movement device using the first calibration matrix and multiple second calibration matrices.

[0169] In the specific implementation, the second calculation module 500 is specifically used to: take the average of multiple second calibration matrices obtained by sequentially executing the point matching process, weight calculation process and gradient descent process from the Nth round to the Mth round to obtain a third calibration matrix, where M is greater than N; calculate the product of the first calibration matrix and the third calibration matrix to obtain a final calibration matrix for calibrating the eye movement device.

[0170] In an embodiment of the present invention, the corresponding relationship between gaze behavior and text content is found from the user's reading behavior when using a computer, thereby completing the establishment of an implicit relationship, and then completing the implicit calibration of the eye movement device, streamlining the calibration process of the eye movement device, and not affecting the user's normal use of the computer during calibration.

[0171] In some embodiments, the process of executing the point matching process by the execution module 400 includes:

[0172] Matching the eye movement coordinate points included in the uncalibrated eye movement coordinate set with the text unit coordinates included in the text unit coordinate set and the edge unit coordinates included in the edge unit coordinate set, respectively, to obtain a point pair set;

[0173] The point pair set includes multiple groups of point pairs, and the point pairs include eye movement coordinate points and text unit coordinates that belong to the same row and are closest to each other, or the point pairs include eye movement coordinate points and edge unit coordinates that are closest to each other.

[0174] In some other embodiments, the process of executing the weight calculation process by the execution module 400 includes: determining the weight of each group of point pairs in the point pair set.

[0175] Specifically, for each group of point pairs in the point pair set, if the point pair contains eye movement coordinate points and text unit coordinates, and the text content corresponding to the text unit coordinates contained in the point pair is not punctuation, calculate the gaze time estimate of the text unit coordinates contained in the point pair.

[0176] The weight of the point pair is calculated using the gaze density of the eye movement coordinate points contained in the point pair and the gaze time estimation of the text unit coordinates.

[0177] If the point pair includes an eye movement coordinate point and a text unit coordinate, and the text content corresponding to the text unit coordinate included in the point pair is a punctuation mark, the weight of the point pair is determined to be a first constant, and the first constant is a negative number.

[0178] If the point pair includes an eye movement coordinate point and an edge unit coordinate, the weight of the point pair is determined to be a second constant, and the second constant is a negative number.

[0179] In some other embodiments, the process of executing the gradient descent process by the execution module 400 includes:

[0180] According to the weights of the eye movement coordinate points, text unit coordinates, edge unit coordinates and point pairs, a gradient descent solution is performed with the minimum weighted distance from the eye movement coordinate points to the text unit coordinates or the edge unit coordinates after moving through the second calibration matrix as the optimization goal to obtain the second calibration matrix; the second calibration matrix is ​​applied to the eye movement coordinate points.

[0181] Preferably, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor; the memory is used to store computer programs; the processor is used to execute computer programs, specifically for implementing the reading-based eye movement device calibration method provided in the above method embodiment.

[0182] Preferably, an embodiment of the present invention further provides a computer storage medium for storing a computer program, which, when executed, is specifically used to implement the reading-based eye movement device calibration method provided in the above method embodiment.

[0183] In summary, the embodiments of the present invention provide a method, system and related equipment for calibrating an eye movement device based on reading, which finds the corresponding relationship between gaze behavior and text content from the user's reading behavior when using a computer, thereby establishing an implicit relationship, and then completing the implicit calibration of the eye movement device, streamlining the calibration process of the eye movement device, and does not affect the user's normal use of the computer during calibration.

[0184] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0185] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0186] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for calibrating an eye movement device based on reading, characterized in that: The method comprises: Collecting an uncalibrated eye movement coordinate set when a user wears an eye movement device to read a specified text, and obtaining a text unit coordinate set and an edge unit coordinate set; Calculating the gaze density of the eye movement coordinate points included in the uncalibrated eye movement coordinate set; Center-aligning the eye movement coordinate point with the text unit coordinates included in the text unit coordinate set to obtain a first calibration matrix; According to the uncalibrated eye movement coordinate set, the text unit coordinate set, the edge unit coordinate set and the gaze density, sequentially execute a point matching process, a weight calculation process and a gradient descent process and repeat the process for multiple rounds to obtain multiple second calibration matrices; The first calibration matrix and a plurality of the second calibration matrices are used to calculate a final calibration matrix for calibrating the eye movement device.

2. The method according to claim 1, characterized in that The process of executing the point matching process includes: Matching the eye movement coordinate points included in the uncalibrated eye movement coordinate set with the text unit coordinates included in the text unit coordinate set and the edge unit coordinates included in the edge unit coordinate set, respectively, to obtain a point pair set; The point pair set includes multiple groups of point pairs, and the point pairs include the eye movement coordinate points and the text unit coordinates that belong to the same row and are closest to each other, or the point pairs include the eye movement coordinate points and the edge unit coordinates that are closest to each other.

3. The method according to claim 2, characterized in that The process of executing the weight calculation process includes: The weight of each group of the point pairs in the point pair set is determined.

4. The method according to claim 3, characterized in that Determining the weight of each group of the point pairs in the point pair set includes: For each group of the point pairs in the point pair set, if the point pairs include the eye movement coordinate points and the text unit coordinates, and the text content corresponding to the text unit coordinates included in the point pairs is not a punctuation, calculating the fixation time estimation of the text unit coordinates included in the point pairs; The weight of the point pair is calculated by using the gaze density of the eye movement coordinate point contained in the point pair and the gaze time estimation of the text unit coordinate; If the point pair includes the eye movement coordinate point and the text unit coordinate, and the text content corresponding to the text unit coordinate included in the point pair is a punctuation mark, determining the weight of the point pair to be a first constant, and the first constant is a negative number; If the point pair includes the eye movement coordinate point and the edge unit coordinate, the weight of the point pair is determined to be a second constant, and the second constant is a negative number.

5. The method according to claim 3 or 4, characterized in that: The process of executing the gradient descent procedure includes: According to the eye movement coordinate point, the text unit coordinate, the edge unit coordinate and the weight of the point pair, a gradient descent solution is performed with the minimum weighted distance from the eye movement coordinate point to the text unit coordinate or the edge unit coordinate after being moved through the second calibration matrix as the optimization goal, to obtain the second calibration matrix; Applying the second calibration matrix to the eye movement coordinate points.

6. The method according to any one of claims 1 to 4, characterized in that: Calculating a final calibration matrix for calibrating the eye movement device using the first calibration matrix and a plurality of the second calibration matrices includes: averaging a plurality of the second calibration matrices obtained by sequentially executing the point matching process, the weight calculation process, and the gradient descent process from the Nth round to the Mth round to obtain a third calibration matrix, where M is greater than N; The product of the first calibration matrix and the third calibration matrix is ​​calculated to obtain a final calibration matrix for calibrating the eye movement device.

7. A reading-based eye movement device calibration system, characterized in that: The system comprises: A collection module, used to collect an uncalibrated eye movement coordinate set when a user wears an eye movement device to read a specified text, and to obtain a text unit coordinate set and an edge unit coordinate set; A first calculation module, configured to calculate the gaze density of the eye movement coordinate points included in the uncalibrated eye movement coordinate set; an alignment module, used for center-aligning the eye movement coordinate point with the text unit coordinates included in the text unit coordinate set to obtain a first calibration matrix; An execution module, configured to sequentially execute a point matching process, a weight calculation process, and a gradient descent process according to the uncalibrated eye movement coordinate set, the text unit coordinate set, the edge unit coordinate set, and the gaze density, and repeat the execution for multiple rounds to obtain multiple second calibration matrices; The second calculation module is used to calculate a final calibration matrix for calibrating the eye movement device by using the first calibration matrix and a plurality of the second calibration matrices.

8. The system according to claim 7, characterized in that The process of the execution module executing the point matching process includes: Matching the eye movement coordinate points included in the uncalibrated eye movement coordinate set with the text unit coordinates included in the text unit coordinate set and the edge unit coordinates included in the edge unit coordinate set, respectively, to obtain a point pair set; The point pair set includes multiple groups of point pairs, and the point pairs include the eye movement coordinate points and the text unit coordinates that belong to the same row and are closest to each other, or the point pairs include the eye movement coordinate points and the edge unit coordinates that are closest to each other.

9. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program, specifically to implement the reading-based eye movement device calibration method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: Used to store a computer program, which, when executed, is specifically used to implement the reading-based eye movement device calibration method as described in any one of claims 1 to 6.

Images