Information processing apparatus, information processing method, and program

Abstract

To provide a mechanism capable of improving accuracy of gaze determination.SOLUTION: An information processing device including a correction coefficient calculation unit configured to calculate a correction coefficient corresponding to an error between first coordinates which are coordinates of a gaze point of a user on a display screen and second coordinates which are coordinates of the gaze point of the user estimated on the basis of line-of-sight data of the user, and a determination unit configured to determine whether or not the user has gazed at content on the basis of a relationship between a gaze area obtained by correcting a display area in which the content is displayed on the display screen using the correction coefficient and the second coordinates.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, many devices that visually output information have been used, such as ATMs (Automatic Teller Machines), PCs (Personal Computers), smartphones, tablet terminals, etc. In such devices, gaze determination may be performed to determine whether or not a user has viewed displayed content.

[0003] Regarding gaze determination, for example, Patent Document 1 discloses a technology in which the coordinates of a gaze point are estimated based on the user's line of sight, and the number of characters the user gazed at is calculated by determining that the user gazed at a character corresponding to the estimated gaze point. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-182083 Summary of the Invention [Problem to be solved by the invention]

[0005] As a preliminary step to gaze determination, calibration is often performed to match the coordinates of the user's actual gaze point with the coordinates of the gaze point estimated based on the user's line of sight. However, an error may occur between the coordinates of the actual gaze point and the coordinates of the estimated gaze point due to factors such as the user wearing glasses. When such an error occurs, the accuracy of gaze determination decreases.

[0006] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a mechanism that can further improve the accuracy of gaze determination. [Means for solving the problem]

[0007] In order to solve the above problem, according to one aspect of the present invention, there is provided an information processing device comprising: a correction coefficient calculation unit that calculates a correction coefficient corresponding to the error between a first coordinate, which is the coordinate of a user's gaze point on a display screen, and a second coordinate, which is the coordinate of the user's gaze point estimated based on the user's gaze data; and a determination unit that determines whether the user has gazed at the content based on the relationship between the gaze area obtained by correcting the display area in which content is displayed on the display screen using the correction coefficient and the second coordinate.

[0008] The gaze area may encompass the display area and be larger than the display area.

[0009] The difference between the display area and the gaze area may be larger as the error between the first coordinates and the second coordinates is larger.

[0010] The correction coefficient calculation unit may calculate the correction coefficient by acquiring, for one or more targets, and statistically processing the error between the coordinates of the target displayed on the display screen and the coordinates of the user's gaze point estimated based on the gaze data obtained when the user gazes at the target.

[0011] The second coordinates may be post-calibration coordinates.

[0012] The information processing device may further include a sentence dividing unit that divides a sentence into a plurality of phrases as the plurality of contents, and a placement unit that determines a plurality of display areas in which the plurality of phrases are displayed on the display screen, and the determination unit may determine whether the user has gazed at each of the plurality of phrases based on the relationship between the gaze area obtained by correcting the display area using the correction coefficient and the second coordinates.

[0013] The determination unit may determine whether the user has gazed at the first segment of the sentence being displayed or the segment immediately following the segment that has already been determined to have been gazed at by the user.

[0014] The arrangement unit may determine each of the plurality of display areas based on the correction coefficient.

[0015] The interval between the two display areas corresponding to the two adjacent phrases may be longer as the error between the first coordinates and the second coordinates increases.

[0016] The information processing device may further include a display control unit that controls display so that the phrases with high importance are displayed in a different display mode from the phrases with low importance.

[0017] After it is determined that the user has gazed at the phrase, the display control unit may control a display mode of the phrase according to the importance of the phrase.

[0018] In addition, in order to solve the above problem, according to another aspect of the present invention, there is provided an information processing method executed by a computer, which includes calculating a correction coefficient corresponding to the error between a first coordinate, which is the coordinate of a user's gaze point on a display screen, and a second coordinate, which is the coordinate of the user's gaze point estimated based on the user's gaze data, and determining whether the user has gazed at the content based on the relationship between the gaze area obtained by correcting the display area in which content is displayed on the display screen using the correction coefficient and the second coordinate.

[0019] In addition, in order to solve the above problem, according to another aspect of the present invention, a program is provided to cause a computer to function as: a correction coefficient calculation unit that calculates a correction coefficient corresponding to the error between a first coordinate, which is the coordinate of a user's gaze point on a display screen, and a second coordinate, which is the coordinate of the user's gaze point estimated based on the user's gaze data; and a determination unit that determines whether the user has gazed at the content based on the relationship between the gaze area obtained by correcting the display area in which content is displayed on the display screen using the correction coefficient and the second coordinate. [Effects of the Invention]

[0020] As described above, according to the present invention, a mechanism capable of further improving the accuracy of gaze determination is provided. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram illustrating an overview of a system 1 according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of the configuration of a system 1 according to the present embodiment. [Figure 3] FIG. 10 is a diagram for explaining an example of a method for calculating a correction coefficient according to the present embodiment. [Figure 4] FIG. 10 is a diagram for explaining an example of a method for calculating a correction coefficient according to the present embodiment. [Figure 5] FIG. 10 is a diagram showing a specific example of the arrangement of phrase areas according to the embodiment. [Figure 6] FIG. 10 is a diagram showing a specific example of the arrangement of gaze areas according to the present embodiment. [Figure 7] 10 is a flowchart showing an example of the flow of processing executed by the system 1 according to the present embodiment. [Figure 8] FIG. 2 is a block diagram showing an example of a hardware configuration of the information processing device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0023] <1. Overview> 1 is a diagram illustrating an overview of a system 1 according to an embodiment of the present invention. As shown in FIG. 1, the system 1 according to this embodiment includes a display device 10 and a gaze detection device 20.

[0024] The display device 10 is a device that visually outputs information. In the example shown in Fig. 1, the display device 10 displays a display screen D including a sentence such as "Please use the driver (No. 1)..."

[0025] The gaze detection device 20 is a device that detects information related to the gaze of the user U (hereinafter also referred to as gaze data). The system 1 can estimate the coordinates of the gaze point of the user U on the display screen D based on the gaze data. In the example shown in FIG. 1, the gaze detection device 20 is installed below the display device 10, but the installation location is not limited to the below the display device 10 and can be any location.

[0026] The coordinates of the actual gaze point of the user U on the display screen D are also referred to as actual coordinates below. The actual coordinates of the gaze point of the user U correspond to the first coordinates.

[0027] On the other hand, the coordinates of the gaze point of the user U on the display screen D estimated by the system 1 will also be referred to as estimated coordinates hereinafter. The estimated coordinates of the gaze point of the user U correspond to the second coordinates.

[0028] These coordinates may be two-dimensional coordinates with an arbitrary point on the display screen D as the origin.

[0029] The system 1 according to this embodiment separates the text to be displayed into phrases such as "Driver (No. 1)," "use," and "please," and displays the text with spaces between the phrases. This configuration improves the readability of long texts such as manuals or contracts when the user U views them, and helps the user U understand the meaning of the text.

[0030] Furthermore, the system 1 according to this embodiment performs calibration and then calculates a correction coefficient corresponding to the error between the actual coordinates and estimated coordinates of the gaze point of the user U that remains after the calibration. The system 1 then determines whether the user U gazed at a phrase based on whether the user U gazed at an area (gaze area, described later) obtained by expanding an area where a phrase is displayed (phrase area, described later) according to the correction coefficient. This configuration makes it possible to suppress a decrease in the accuracy of gaze determination due to an error that remains after the calibration, and to accurately determine whether a phrase was gazed at.

[0031] Furthermore, the system 1 highlights phrases that contain important characters in the text to be displayed. This configuration makes it possible to impress on the user the important points that the author of the text wants to convey, thereby reducing the likelihood of overlooking important points.

[0032] In this way, the system 1 according to this embodiment can further improve the accuracy of gaze determination and can also provide suitable support for the user U in browsing text.

[0033] <2.Configuration example> Fig. 2 is a block diagram showing an example of the configuration of a system 1 according to this embodiment. As shown in Fig. 2, the system 1 includes an input device 30, an information processing device 40, and a server 50, in addition to the display device 10 and the gaze detection device 20 described with reference to Fig. 1. These devices can be connected via a wired or wireless network.

[0034] (Display device 10) The display device 10 is a device that visually outputs information. The display device 10 is, for example, a display. The display device 10 displays a display screen D under the control of the information processing device 40.

[0035] (Gaze detection device 20) The gaze detection device 20 is a device that detects gaze data of the user U. The gaze detection device 20 may be a sensor such as an infrared sensor, an infrared camera, or a visible light camera. The gaze detection device 20 outputs data detected regarding the eyes of the user U to the information processing device 40 as gaze data.

[0036] (input device 30) The input device 30 accepts information input from the user U. The input device 30 may be, for example, a mouse, a keyboard, or a touch panel. Alternatively, the input device 30 may be a microphone and may accept voice input. Furthermore, the gaze detection device 20 may also serve as the input device 30 and may accept gaze input.

[0037] For example, the input device 30 accepts an operation to select a sentence to be displayed from one or more sentences stored in the server 50. Then, the display device 10 displays a display screen D including the sentence to be displayed selected by the user U.

[0038] (Server 50) The server 50 includes a text database (hereinafter also referred to as DB) 51 and an emphasis character DB 52.

[0039] (Text DB51) The sentence DB 51 stores data indicating one or more sentences. An example of the data stored in the sentence DB 51 is shown in Table 1.

[0040] [Table 1]

[0041] "Page" in Table 1 is the page number on which the text is displayed. In other words, the text to be displayed can span multiple pages.

[0042] (emphasized character DB52) The emphasis character DB 52 stores data indicating characters to be emphasized. An example of the data stored in the emphasis character DB 52 is shown in Table 2.

[0043] [Table 2]

[0044] As shown in Table 2, the emphasized character DB 52 may store characters to be emphasized in association with character colors as information specifying a method for emphasizing the characters to be emphasized. "index" is a symbol for uniquely distinguishing characters to be emphasized.

[0045] The data in the highlighted character DB 52 may be registered manually. Alternatively, the data in the highlighted character DB 52 may be registered automatically using morphological analysis or machine learning. An example of a method for automatically registering data in the highlighted character DB 52 will be described below.

[0046] First, the server 50 performs a morphological analysis of the sentences stored in the sentence DB 51 and divides the sentences into morphemes (hereinafter also referred to as words). Note that morphological analysis refers to dividing a sentence into morphemes, which are the smallest units with meaning.

[0047] Second, the server 50 uses a word embedding model to represent each morpheme segmented from the sentence as a vector in a multidimensional space (hereinafter also referred to as a word vector). The server 50 also represents pre-registered keywords as word vectors. Note that, for example, Word2vec can be adopted as the word embedding model.

[0048] Third, the server 50 calculates the similarity between the keyword and each morpheme. For example, the similarity may be the cosine similarity between the word vector of the keyword and the word vector of the morpheme.

[0049] Fourth, the server 50 extracts morphemes whose similarity to the keyword is equal to or greater than a predetermined threshold as morphemes related to the keyword, and registers them in the emphasized character DB 52 as characters to be emphasized.

[0050] The above describes an example of a method for automatically registering data in the emphasized character DB 52. Below, a specific example of a method for automatically registering data in the emphasized character DB 52 will be described using "Sakura wa uchikushii" (cherry blossoms are beautiful) as an example sentence and "hana (flowers)" as an example keyword.

[0051] First, the server 50 performs morphological analysis to divide the sentence "Sakura wa utsukushii" into a plurality of morphemes such as "sakura," "wa," and "utsukushii."

[0052] Second, the server 50 applies a word embedding model to create a word vector for "sakura" [1,0,0], a word vector for "wa" [0,1,0], a word vector for "utsui" [0.2,0,0.7], and a word vector for "hana" [0.9,0,0.1].

[0053] Third, the server 50 calculates a cosine similarity of 0.9 between the word vectors of "sakura" and "hana", a cosine similarity of 0 between the word vectors of "wa" and "hana", and a cosine similarity of 0.25 between the word vectors of "utsui" and "hana".

[0054] Fourth, the server 50 extracts "sakura" which has a cosine similarity with the keyword "hana" that is equal to or greater than the threshold value of 0.8, and registers it in the emphasized character DB 52 as a character to be emphasized.

[0055] (information processing device 40) The information processing device 40 is a device that controls the overall operation of the system 1. Specifically, the information processing device 40 displays a sentence to be displayed on the display device 10 in a segmented state, performs gaze determination for each phrase based on gaze data obtained by the gaze detection device 20, and controls processing for emphasizing keywords. As shown in FIG. 2 , the information processing device 40 includes a calibration unit 41, a gaze point estimation unit 42, a correction coefficient calculation unit 43, a parameter DB 44, a sentence segmentation unit 45, an arrangement unit 46, a phrase DB 47, a determination unit 48, and a display control unit 49.

[0056] (Calibration section 41) The calibration unit 41 performs calibration based on the gaze data output from the gaze detection device 20. Then, the calibration unit 41 outputs the calibration data to the gaze point estimation unit .

[0057] For example, the calibration unit 41 may construct an anatomical three-dimensional (3D) eye model of the user U based on information (such as data on the position of the cornea and fovea) about the shape of the eyeball, refraction of light, and reflection characteristics of the user U, which is included in the gaze data. Then, the calibration unit 41 may output the deviation between the optical axis and the visual axis in the 3D eye model of the user U to the gaze point estimation unit 42 as calibration data.

[0058] It should be noted that known techniques may be employed for the calibration. As an example, a calibration method may be employed in which a plurality of calibration targets are displayed on the display screen D and the user U gazes at the calibration targets. As another example, a calibration method may be employed in which the user U wears a wearable eye tracker.

[0059] (Gaze point estimation unit 42) The gaze point estimation unit 42 estimates estimated coordinates of the gaze point of the user U on the display screen D based on the gaze data output from the gaze detection device 20. In particular, the gaze point estimation unit 42 estimates estimated coordinates of the gaze point of the user U on the display screen D based on the gaze data output from the gaze detection device 20 and the calibration data output from the calibration unit 41. In other words, the estimated coordinates of the gaze point of the user U are coordinates after calibration.

[0060] The gaze point estimation unit 42 outputs the estimated coordinates of the gaze point of the user U to the correction coefficient calculation unit 43 and the determination unit 48.

[0061] (Correction coefficient calculation unit 43) The correction coefficient calculation unit 43 calculates a correction coefficient corresponding to the error between the actual coordinates of the gaze point of the user U on the display screen D and the estimated coordinates of the gaze point of the user U estimated based on the line-of-sight data of the user U. The correction coefficient calculation unit 43 outputs the calculated correction coefficient to the parameter DB 44 and stores it in the parameter DB 44.

[0062] Specifically, the correction coefficient calculation unit 43 calculates a correction coefficient by acquiring, for one or more targets, the error between the coordinates of the target displayed on the display screen D and the estimated coordinates of the user U's point of gaze, which is estimated based on gaze data obtained when the user U gazes at the target, and performing statistical processing. The target may be, for example, a black dot, and may be displayed at any coordinate on the display screen D. The error between the target coordinates and the estimated coordinates of the user U's point of gaze may be, for example, a distance or an angle. The statistical processing may be an average value, a weighted average value, or a median value. This configuration makes it possible to calculate a correction coefficient that can appropriately correct the error between the actual coordinates and the estimated coordinates of the user U's point of gaze. An example of a method for calculating the correction coefficient will be described below with reference to FIGS. 3 and 4.

[0063] 3 and 4 are diagrams illustrating an example of a method for calculating a correction coefficient according to this embodiment. As shown in FIG. 3, the correction coefficient calculation unit 43 causes the display device 10 to display a display screen D including targets P (P1 to P4). The coordinates of the targets P1 to P4 may be random. The user U gazes at the targets P1 to P4 displayed on the display device 10 in order. Then, the correction coefficient calculation unit 43 obtains, from the gaze point estimation unit 42, estimated coordinates of the gaze point of the user U, which are estimated based on gaze data obtained by the gaze detection device 20 when the user U gazes at each of the targets P1 to P4.

[0064] For example, as shown in FIG. 4, it is assumed that gaze data indicating four gaze points Pos1 to Pos4 are obtained as gaze points when a user U gazes at a target P1. Then, the estimated coordinates of these gaze points Pos1 to Pos4 when the coordinates of the target P1 are set as the origin are expressed as (x P1-1 ,y P1-1 ), (x P1-2 ,y P1-2 ), (x P1-3 ,y P1-3 ), and (x P1-4 ,y P1-4 ) is assumed to be obtained.

[0065] The correction coefficient calculation unit 43 calculates the error between the coordinates of the targets P1 to P4 displayed on the display screen D and the estimated coordinates of the gaze points when the user U gazes at these targets P1 to P4. For example, as the error regarding the target P1, the distance r P1-1 ~r P1-4 The arithmetic mean of r P1 is calculated by the following formula:

[0066]

number

[0067] Then, the correction coefficient calculation unit 43 calculates the distance r between the target P1 to P4 and the estimated coordinates of the gaze point of the user U, which is calculated for each of the targets P1 to P4. P1 ~r P4 The arithmetic mean of raverage is calculated using the following formula:

[0068]

number

[0069] The correction coefficient calculation unit 43 calculates, for example, r average is adopted as the correction factor.

[0070] Although the above describes an example in which targets P1 to P4 are displayed simultaneously, the present invention is not limited to this example. The correction coefficient calculation unit 43 may display any one of targets P1 to P4 on the display device 10. The correction coefficient calculation unit 43 may switch the target P to be displayed on the display device 10 every time estimated coordinates of the gaze point of the user U required for calculating the correction coefficient are obtained.

[0071] (Parameter DB44) The parameter DB 44 stores various parameters related to the operation of the information processing device 40. An example of the parameters stored in the parameter DB 44 is shown in Table 3 below.

[0072] [Table 3]

[0073] The correction coefficient α is a correction coefficient calculated by the correction coefficient calculation unit 43. The spacing d between phrase areas is the spacing between phrase areas when a sentence is displayed on the display device 10. The sentence area indicates the range of the display screen D in which a sentence can be displayed. The gaze determination threshold is a threshold that is compared with the time spent gazing at a phrase, and is used to determine whether or not the user U has gazed at a phrase.

[0074] (Sentence division part 45) The sentence segmentation unit 45 segments a sentence into a plurality of phrases. The sentence segmentation unit 45 acquires a sentence to be displayed from the sentence DB 51 and segments the acquired sentence into a plurality of phrases. The sentence segmentation unit 45 then stores information indicating the created plurality of phrases in the phrase DB 47.

[0075] The sentence segmentation unit 45 obtains a plurality of phrases by dividing the sentence at appropriate delimiter positions using morphological analysis or machine learning. An example of processing by the sentence segmentation unit 45 will be described below.

[0076] First, the sentence dividing unit 45 performs a morphological analysis of the sentence and divides the sentence into morphemes (hereinafter also referred to as words).

[0077] Second, the sentence segmentation unit 45 analyzes the dependency relationship for each word. Dependency is a concept that refers to the relationship between words in a sentence. By analyzing the dependency relationship, the relationship between words becomes clear, making it possible to understand the structure of the sentence.

[0078] Third, the sentence segmentation unit 45 creates phrases based on dependency relationships. A phrase is a collection of one main word and words related to that main word. Therefore, the sentence segmentation unit 45 creates phrases by combining multiple words that are determined to have a dependency relationship or whose relationship is equal to or exceeds a predetermined threshold. With this configuration, phrases that are easy for the user U to read are created.

[0079] Fourth, the sentence segmentation unit 45 assigns numbers to the created phrases. For example, the sentence segmentation unit 45 assigns numbers to phrases that indicate the order in which the phrases appear in the sentence (for example, "index" shown in Table 4).

[0080] Fifth, the sentence segmentation unit 45 outputs information associating the created phrases with the numbers assigned to the phrases.

[0081] The above describes an example of the processing by the sentence segmentation unit 45. Below, a specific example of the processing by the sentence segmentation unit 45 will be described using the example sentence "I'm going to school."

[0082] First, the sentence dividing unit 45 performs morphological analysis to divide the sentence "I am going to school" into a plurality of morphemes such as "I," "wa," "gakkou," "e," "iku," and "masu."

[0083] Second, the sentence segmentation unit 45 analyzes the dependency relationship for each word. As a result, it is analyzed that "watashi" is related to "wa", "gakko" is related to "he", and "yuki" is related to "masu".

[0084] Third, the sentence dividing unit 45 combines a plurality of words that have been determined to have a dependency relationship, and creates phrases such as "I," "I'm going to school," and "I'm going."

[0085] Fourth, the sentence segmentation unit 45 assigns 1 to "I"; 2 to "to school"; and 3 to "I'm going."

[0086] Fifth, the sentence segmentation unit 45 outputs "1" "I am," "2" "I am going to school," and "3" "I am going."

[0087] (Arrangement section 46) The placement unit 46 refers to the phrase DB 47 and determines various placements on the display screen D for the phrases divided by the sentence dividing unit 45.

[0088] More specifically, the placement unit 46 determines a plurality of display areas (hereinafter also referred to as phrase areas) in which a plurality of phrases constituting a sentence to be displayed are displayed on the display screen D. More specifically, the placement unit 46 determines a plurality of phrase areas in which a plurality of phrases obtained by dividing the sentence to be displayed by the sentence dividing unit 45 are displayed, based on various parameters stored in the parameter DB 44. The phrase areas may be rectangular, and phrases are displayed within the phrase areas. For example, the placement unit 46 determines the coordinates of the center of the phrase area and the coordinates of the four corners of the phrase area for each phrase.

[0089] The phrase is an example of content in this embodiment, and the phrase area is an example of a content display area in this embodiment.

[0090] A specific example of the arrangement of phrase areas will be described below with reference to Fig. 5. In this specific example, the sentence to be displayed, "Please use the screwdriver (No. 1) to fasten the S1 screw to positions A and C," is divided by the sentence dividing unit 45 into seven phrases: "Please use the screwdriver (No. 1)," "fasten the S1 screw," "at positions A and C," and "please."

[0091] 5 is a diagram showing a specific example of the placement of phrase areas according to this embodiment. As shown in FIG. 5, the placement unit 46 determines phrase areas W (W1 to W7) of seven phrases that make up the sentence to be displayed within a sentence area T specified by the parameters xleft, xright, ytop, and ybottom stored in the parameter DB 44. For example, for phrase area W1, the coordinates of the center (x1, y1) and the coordinates of the upper right corner (x tr1 ,y tr1 ), the coordinates of the upper left corner (x tl1 ,y tl1 ), the coordinate of the bottom right corner (x br1 ,y br1 ), and the coordinate of the bottom left corner (x bl1 ,y bl1 The distance between adjacent phrase areas W is set to the distance d stored in the parameter DB 44.

[0092] Next, the placement unit 46 determines the gaze area by correcting the phrase area on the display screen D using the correction coefficient α. One condition is that the user U's gaze point stays in the gaze area of ​​a phrase, and it is determined that the user U has gazed at that phrase. Here, the gaze area is defined for each phrase, so gaze determination is performed for each phrase. In an environment where an error may occur between the actual coordinates and estimated coordinates of the user U's gaze point, it may be difficult to perform gaze determination for each character. In this regard, by forming a gaze area for each phrase and performing gaze determination, it is possible to suppress a decrease in the accuracy of gaze determination.

[0093] Here, the gaze area encompasses the phrase area and is larger than the phrase area. With this configuration, even if an error occurs between the actual coordinates and estimated coordinates of the user U's gaze point, it is possible to prevent the user from being erroneously determined not to be gazing at a phrase despite actually gazing at it. However, the gaze area is formed separate from other gaze areas (i.e., so as not to overlap). With this configuration, the gaze area can be formed by utilizing the gaps between phrases created by word segmentation, making it possible to both improve the ease of understanding the meaning of a sentence through word segmentation and improve the accuracy of gaze determination.

[0094] The difference (for example, area difference) between the phrase area and the gaze area may be larger as the error between the actual coordinates and estimated coordinates of the gaze point of the user U is larger, and may be smaller as the error is smaller. average takes a larger value as the error between the actual coordinates and estimated coordinates of the gaze point of user U increases. When such a correction coefficient α is adopted, the placement unit 46 determines as the gaze area an area that is a larger expansion of the phrase area as the correction coefficient α increases, and determines as the gaze area an area that is a smaller expansion of the phrase area as the correction coefficient α decreases. With this configuration, it is possible to determine a gaze area that is large enough to prevent a decrease in the estimation accuracy of gaze determination.

[0095] A specific example of the arrangement of the gaze areas will be described below with reference to Fig. 6. Here, a specific example of the arrangement of the gaze areas corresponding to the phrase area W1 shown in Fig. 5 will be described.

[0096] FIG. 6 is a diagram showing a specific example of the arrangement of the gaze areas according to this embodiment. As shown in FIG. 6, the arrangement unit 46 may determine the gaze area G1 by expanding the phrase area W1 in all directions by the correction coefficient α. That is, the arrangement unit 46 determines the coordinates (x tr1 ,y tr1 ) is moved right and up by the correction coefficient α, and the coordinates are the coordinates (x´ tr1 ,y´ tr1The coordinates of the other corners of the gaze area G1 are determined in a similar manner. Note that the center coordinates of the phrase area W1 and the center coordinates of the gaze area G1 are the same.

[0097] Note that the example described with reference to Figure 6 is merely one example, and various methods for determining the gaze area are conceivable. As one example, an area obtained by expanding the phrase area only in either the vertical or horizontal direction may be determined as the gaze area. As another example, the longer the phrase, the longer the area in the horizontal direction the longer the phrase area is extended, and the more the gaze area may be determined as the gaze area.

[0098] The arrangement unit 46 stores information indicating the determined phrase area and gaze area in the phrase DB 47.

[0099] (Clause DB47) The phrase DB 47 stores data for each phrase. Specifically, the phrase DB 47 stores information indicating phrases output from the sentence segmentation unit 45. Furthermore, the phrase DB 47 stores information indicating phrase areas and gaze areas determined by the placement unit 46. Furthermore, the phrase DB 47 stores gaze determination results for each phrase.

[0100] An example of data stored in the clause DB 47 is shown in Table 4 below.

[0101] [Table 4]

[0102] "Page" in Table 4 indicates the page on which the phrase is displayed. "Index" is a symbol for uniquely distinguishing phrases within a page. "Phrase" is a phrase divided by the sentence dividing unit 45. "Center coordinates" are the center coordinates of the phrase area and the gaze area. "Phrase area coordinates" are the coordinates of the four corners of the phrase area determined by the placement unit 46. "Gaze area coordinates" are the coordinates of the four corners of the phrase gaze area determined by the placement unit 46. "Gaze determination result" is the determination result by the determination unit 48 as to whether or not a phrase has been gazed upon, and is "True" if the phrase has been gazed upon, and is "False" if the phrase has not yet been gazed upon.

[0103] (Judgment section 48) The determination unit 48 determines whether or not the user U has gazed at the phrase.

[0104] In detail, the determination unit 48 determines whether the user U has gazed at each of the multiple phrases based on the relationship between the gaze area obtained by correcting the phrase area using the correction coefficient α and the estimated coordinates of the user U's gaze point.

[0105] As an example, when the estimated coordinates of the user U's point of gaze are within the gaze area, the determination unit 48 may determine that the user U has gazed at a phrase corresponding to the gaze area. On the other hand, when the estimated coordinates of the user U's point of gaze have never entered the gaze area, the determination unit 48 may determine that the user U has not gazed at a phrase corresponding to the gaze area.

[0106] As another example, the determination unit 48 determines that the user U has gazed at a phrase corresponding to the gaze area when the time during which the estimated coordinates of the user U's gaze point stayed inside the gaze area is equal to or greater than the gaze determination threshold stored in the parameter DB 44. On the other hand, the determination unit 48 determines that the user U has not gazed at a phrase corresponding to the gaze area when the time during which the estimated coordinates of the user U's gaze point stayed inside the gaze area is less than the gaze determination threshold stored in the parameter DB 44.

[0107] Here, the determination unit 48 may determine whether or not the user U has gazed at the first segment of the sentence being displayed, or the segment immediately following a segment that has already been determined to have been gazed at by the user U. In particular, immediately after the display of the sentence to be displayed is started, the determination unit 48 may determine whether or not the user U has gazed at only the first segment. After it has been determined that the user U has gazed at the first segment, the determination unit 48 may determine whether or not the user U has gazed at the segment immediately preceding the first segment that has been determined to have not yet been gazed at by the user U. This configuration makes it possible to prevent skipping over a sentence by starting to read in the middle.

[0108] The determination unit 48 stores the determination result in the phrase DB 47. For example, the determination unit 48 rewrites the gaze determination result of the phrase that is determined to have been gazed upon by the user U from "False" to "True."

[0109] (Display control unit 49) The display control unit 49 controls the display of the display screen D by the display device 10. In detail, the display control unit 49 refers to the phrase DB 47, and arranges and renders each of the multiple phrases included in the sentence to be displayed in the phrase area determined by the arrangement unit 46, and causes the display device 10 to display the phrases.

[0110] Furthermore, the display control unit 49 controls so that phrases of high importance are displayed in a different display mode from phrases of low importance. Specifically, the display control unit 49 refers to the emphasis character DB 52 and controls the display device 10 so that, of the phrases included in the sentence to be displayed, phrases including characters to be emphasized are displayed in a specified character color (e.g., red). On the other hand, the display control unit 49 controls the display device 10 so that other phrases are displayed in a different character color (e.g., black). This configuration makes it possible to make a strong impression on the user U of phrases including characters of high importance.

[0111] Here, after determining that the user U has gazed at a phrase, the display control unit 49 may control the display mode of the phrase according to the importance of the phrase. As an example, the display control unit 49 may refer to the phrase DB 47, and after determining that the user U has gazed at all phrases displayed on the display screen D, change the display mode of a phrase containing characters with high importance. With this configuration, the user U can initially concentrate on understanding the entire sentence and then recheck the emphasized important parts after finishing reading. As a result, it becomes easier for the user U to grasp the meaning of the sentence.

[0112] Of course, the display control unit 49 may control the display mode of a phrase including characters of high importance without waiting until it is determined that all phrases displayed on the display screen D have been gazed upon by the user U. In other words, the display control unit 49 may control the display mode of a phrase in accordance with the importance of the phrase immediately after it is determined that the phrase has been gazed upon by the user U.

[0113] <3. Processing flow> FIG. 7 is a flowchart showing an example of the flow of processing executed by the system 1 according to this embodiment.

[0114] First, the system 1 selects a sentence to be displayed (step S102). For example, the input device 30 accepts an operation by the user U to select a sentence to be displayed from among a plurality of sentences stored in the sentence DB 51.

[0115] Next, the sentence dividing unit 45 divides the sentence to be displayed into a plurality of phrases (step S104).The sentence dividing unit 45 then stores information indicating the divided phrases in the phrase DB 47.

[0116] Next, the placement unit 46 determines a phrase area (step S106), and stores information indicating the determined phrase area in the phrase DB 47.

[0117] Next, the calibration unit 41 performs calibration based on the gaze data acquired by the gaze detection device 20 (step S108). Then, the calibration unit 41 outputs the calibration data to the gaze point estimation unit .

[0118] Next, the correction coefficient calculation unit 43 calculates the correction coefficient α (step S110), and stores the calculated correction coefficient α in the parameter DB 44.

[0119] Next, the arrangement unit 46 determines a gaze area by expanding the phrase area stored in the phrase DB 47 based on the correction coefficient α stored in the parameter DB 44 (step S112). Then, the arrangement unit 46 stores information indicating the determined gaze area in the phrase DB 47.

[0120] Next, the display control unit 49 starts displaying the sentence to be displayed (step S114). For example, the display control unit 49 refers to the phrase DB 47, renders a display screen D in which each of the multiple phrases constituting the sentence to be displayed is arranged in a corresponding phrase area, and displays the display screen D on the display device 10. The system 1 also starts detecting gaze data of the user U using the gaze detection device 20. At the same time, the system 1 starts estimating the gaze point of the user U using the gaze point estimation unit 42 based on the detected gaze data and calibration data.

[0121] Next, the determination unit 48 determines whether or not the user U has gazed at a phrase (step S116). Then, the determination unit 48 stores the determination result in the phrase DB 47. That is, the determination unit 48 enters "True" as the gaze determination result in the phrase DB 47 for the phrase that it has determined that the user U has gazed at.

[0122] The process in step S116 is repeated until it is determined that the user U has gazed at the phrase (step S116: NO).

[0123] If it is determined that the user U has gazed at a phrase (step S116: YES), the display control unit 49 determines whether or not the user U has gazed at all phrases in the currently displayed page (step S118). For example, the display control unit 49 determines whether or not the gaze determination results of all phrases included in the currently displayed page and stored in the phrase DB 47 are "True."

[0124] If it is determined that there is a phrase in the currently displayed page that the user U is not gazing at (step S118: NO), the process returns to step S116 again.

[0125] On the other hand, if it is determined that the user U has gazed at all the phrases in the displayed page (step S118: YES), the display control unit 49 highlights important phrases (step S120). For example, the display control unit 49 refers to the highlighted character DB 52 and displays phrases containing characters to be highlighted in red, and displays other phrases in black.

[0126] Next, the display control unit 49 determines whether all pages have been viewed (step S122). For example, the display control unit 49 determines whether the gaze determination results of all phrases on all pages stored in the phrase DB 47 are “True”.

[0127] If it is determined that there is an unviewed page (step S122: NO), the display control unit 49 starts displaying the text of the next page (step S124), after which the process returns to step S116.

[0128] On the other hand, if it is determined that all pages have been viewed (step S122: YES), the process ends.

[0129] <4. Hardware configuration example> Next, the hardware configuration of an information processing device according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of the hardware configuration of an information processing device according to this embodiment. Note that the information processing device 900 shown in Fig. 8 may realize, for example, various components included in the system 1 shown in Fig. 2. Information processing by the system 1 according to this embodiment is realized by cooperation between software and hardware described below.

[0130] As shown in FIG. 8, the information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0131] The CPU 901 functions as an arithmetic processing unit and control unit, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, and parameters that change as appropriate during the execution. These are connected to each other by a host bus 904 consisting of a CPU bus, etc. The CPU 901 may form, for example, the calibration unit 41, the gaze point estimation unit 42, the correction coefficient calculation unit 43, the sentence segmentation unit 45, the placement unit 46, the determination unit 48, or the display control unit 49 shown in FIG. 2 .

[0132] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not necessary to configure the host bus 904, bridge 905, and external bus 906 separately, and these functions may be implemented on a single bus.

[0133] The input device 908 is composed of input means for the user to input information, such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. A user who operates the information processing device 900 can input various data and instruct the information processing device 900 to perform processing operations by operating this input device 908. The input device 908 can form, for example, the input device 30 shown in FIG. 2.

[0134] The output device 909 may include, for example, a display device that outputs visual information, such as a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp. The output device 909 may include an audio output device that outputs auditory information, such as a speaker. The output device 909 may include a tactile presentation device that outputs tactile information, such as an eccentric motor. The output device 909 may form, for example, the display device 10 shown in FIG. 2.

[0135] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded on the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). The storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data. The storage device 910 may form, for example, the parameter DB 44, the phrase DB 47, the sentence DB 51, or the highlighted character DB 52 shown in FIG. 2.

[0136] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network, etc. The communication device 911 may be compatible with either wireless communication or wired communication.

[0137] The above describes an example of a hardware configuration capable of realizing the functions of the information processing device 900 according to this embodiment. Each of the above components may be realized using general-purpose components, or may be realized by hardware specialized for the function of each component. Therefore, the hardware configuration used can be changed as appropriate depending on the technical level at the time of implementing this embodiment.

[0138] <5. Supplementary Information> Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0139] The arrangement unit 46 may determine each of the multiple phrase areas based on the correction coefficient α. Specifically, the arrangement unit 46 may set the distance d between two phrase areas corresponding to two adjacent phrases so that the distance d becomes longer as the error between the actual coordinates and estimated coordinates of the user U's gaze point increases. For example, the phrase DB 47 may set the distance d to a value obtained by multiplying the correction coefficient α by a constant of at least two. This configuration makes it possible to prevent overlapping of gaze areas between adjacent phrases and improve the accuracy of gaze determination.

[0140] The placement unit 46 may determine the gaze area based on the importance of characters included in a phrase. Specifically, the placement unit 46 may determine a narrower gaze area for a phrase including characters of high importance compared to a phrase not including characters of high importance. For example, the placement unit 46 may determine the gaze area by expanding the phrase area vertically and horizontally by the correction coefficient α for phrases including characters of high importance and by twice the correction coefficient α for phrases other than those. With this configuration, the more important a phrase is, the stricter the gaze determination, making it possible to better prevent skipping over it.

[0141] In the above embodiment, an example has been described in which the display device 10 is a display, but the present invention is not limited to such an example. The display device 10 may be a projector or a head mounted display (HMD).

[0142] In the above embodiment, a phrase is given as an example of content, but the present invention is not limited to this example, and the present invention may be applied to other content such as an image.

[0143] In the above embodiment, changing the character color is given as an example of a method for highlighting a phrase, but the present invention is not limited to this example. Other methods that may be adopted include increasing the character size, changing the font, or adding underlines.

[0144] Note that each device described in this specification may be realized as a single device, or some or all of them may be realized as separate devices. As one example, the display device 10 and the input device 30 may be realized as a touch-operable display or touch panel. As another example, the display device 10, the input device 30, and the information processing device 40 may be realized as a tablet terminal, a smartphone, or a desktop PC (Personal Computer). As another example, the information processing device 40 may include a sentence DB 51 and a highlighted character DB 52. As another example, the server 50 may include a parameter DB 44 and a phrase DB 47.

[0145] The series of processes performed by each device described herein may be implemented using software, hardware, or a combination of software and hardware. The software programs may be stored in advance, for example, on a recording medium (more specifically, a non-transitory computer-readable storage medium) internal or external to each device. Each program is loaded into a random access memory (RAM) and executed by a processing circuit such as a central processing unit (CPU). The recording medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. The computer program may be distributed, for example, via a network without using a recording medium. The computer may be an application-specific integrated circuit (ASIC), a general-purpose processor that executes functions by loading a software program, or a computer on a server used in cloud computing. The series of processes performed by each device described herein may be centrally processed by a single computer or distributed across multiple computers. Furthermore, in each of the above embodiments, two or more communication means present in one device may be physically implemented on a single medium.

[0146] Furthermore, the processes described herein using flowcharts or sequence diagrams do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. Furthermore, additional process steps may be employed, and some process steps may be omitted. [Explanation of symbols]

[0147] 1 System 10 Display device 20. Line of sight detection device 30 Input Devices 40 Information processing equipment 41 Calibration section 42 Gaze point estimation unit 43 Correction coefficient calculation unit 44 Parameter DB 45 Sentence division part 46 Placement section 47 Phrase DB 48 Judgment section 49 Display control unit 50 servers 51 Text DB 52 Emphasized character DB

Claims

1. a correction coefficient calculation unit that calculates a correction coefficient corresponding to an error between a first coordinate that is a coordinate of a user's gaze point on a display screen and a second coordinate that is a coordinate of the user's gaze point estimated based on the user's gaze data; a determination unit that determines whether the user has gazed at the content based on a relationship between a gaze area obtained by correcting a display area in which the content is displayed on the display screen using the correction coefficient and the second coordinates; and An information processing device comprising:

2. The gaze area includes the display area and is larger than the display area. The information processing device according to claim 1 .

3. The difference between the display area and the gaze area increases as the error between the first coordinates and the second coordinates increases. The information processing device according to claim 2 .

4. the correction coefficient calculation unit calculates the correction coefficient by acquiring, for one or more targets, an error between the coordinates of the target displayed on the display screen and the coordinates of the user's gaze point estimated based on the gaze data obtained when the user gazes at the target, and performing statistical processing on the error. The information processing device according to claim 1 .

5. the second coordinates are coordinates after calibration; The information processing device according to claim 1 .

6. The information processing device includes: a sentence dividing unit that divides a sentence into a plurality of phrases as the content; an arrangement unit that determines a plurality of display areas in which a plurality of the phrases are to be displayed on the display screen; Furthermore, the determination unit determines whether the user has gazed at each of the plurality of phrases based on a relationship between the gaze area obtained by correcting the display area using the correction coefficient and the second coordinates. The information processing device according to claim 1 .

7. the determination unit determines whether the user has gazed at the first segment of the sentence being displayed or the segment immediately following the segment that has already been determined to have been gazed at by the user; The information processing device according to claim 6 .

8. the placement unit determines each of the plurality of display areas based on the correction coefficient; The information processing device according to claim 6 .

9. the interval between the two display areas corresponding to the two adjacent phrases is longer as the error between the first coordinates and the second coordinates is larger; The information processing device according to claim 8 .

10. The information processing device includes: a display control unit that controls the display of the phrases with high importance in a different display mode from the phrases with low importance; The information processing device according to claim 6 .

11. the display control unit controls a display mode of the phrase according to the importance of the phrase after it is determined that the user has gazed at the phrase. The information processing device according to claim 10.

12. 1. A computer-implemented information processing method, comprising: calculating a correction coefficient corresponding to an error between a first coordinate, which is a coordinate of a user's gaze point on a display screen, and a second coordinate, which is a coordinate of the user's gaze point estimated based on the user's gaze data; determining whether the user has gazed at the content based on a relationship between a gaze area obtained by correcting a display area in which the content is displayed on the display screen using the correction coefficient and the second coordinates; An information processing method including:

13. Computer, a correction coefficient calculation unit that calculates a correction coefficient corresponding to an error between a first coordinate that is a coordinate of a user's gaze point on a display screen and a second coordinate that is a coordinate of the user's gaze point estimated based on the user's gaze data; a determination unit that determines whether the user has gazed at the content based on a relationship between a gaze area obtained by correcting a display area in which the content is displayed on the display screen using the correction coefficient and the second coordinates; and A program to function as a

Images