Correction device, correction method, and correction program

Abstract

To provide a technique capable of improving the detection accuracy of a visual line position.SOLUTION: A correction unit 11 of a correction device 1 acquires respective visual line positions AT of a user U detected by a tablet terminal 2 when the user U imaged by a camera 21 of the tablet terminal 2 gazes at a plurality of target points TG sequentially displayed on a monitor screen 22, and corrects distortion (visual line position deviation) of the visual line positions AT of the user U based on the fact that the camera 21 has imaged the user U from the direction of the edge of the monitor screen 22. The correction unit 11 determines whether the correction succeeds or fails on the basis of a predetermined criterion, calculates an estimated value of the pre-correction visual line position AT corresponding to the failed visual line position using each pre-correction visual line position AT of a plurality of visual line positions that succeeded in correction when there is a visual line position for which the correction failed, and sets the estimated value as the new pre-correction visual line position AT of the failed visual line position.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a correction device, a correction method, and a correction program.

Background Art

[0002] Non-Patent Document 1 discloses a technique for examining a subject's cognitive function by a computer based on the eye movement photographed by a camera.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, as shown in FIG. 1 of Non-Patent Document 1, since the camera is arranged at the edge of the monitor screen, distortion occurs in the line-of-sight position of the subject detected by the computer.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to provide a technique capable of improving the detection accuracy of the line-of-sight position.

Means for Solving the Problems

[0006] The correction device according to one aspect of the present invention acquires each line-of-sight position of the user detected by a computer when the user photographed by a camera gazes at a plurality of target points sequentially displayed within a screen, and includes a correction unit that corrects the distortion of the line-of-sight position of the user based on the fact that the camera photographed the user from a specific direction. The correction unit determines success or failure of the correction based on a predetermined criterion. When there is a line-of-sight position for which the correction has failed, an estimated value of the line-of-sight position before correction related to the failed line-of-sight position is calculated using each line-of-sight position before correction of a plurality of line-of-sight positions for which the correction has succeeded, and the estimated value is set as a new line-of-sight position before correction of the failed line-of-sight position.

[0007] The correction method according to one aspect of the present invention is a correction method performed by a correction device. In the correction method, each line-of-sight position of the user detected by a computer when the user photographed by a camera gazes at a plurality of target points sequentially displayed within a screen is acquired, the distortion of the line-of-sight position of the user based on the fact that the camera photographed the user from a specific direction is corrected, success or failure of the correction is determined based on a predetermined criterion, and when there is a line-of-sight position for which the correction has failed, an estimated value of the line-of-sight position before correction related to the failed line-of-sight position is calculated using each line-of-sight position before correction of a plurality of line-of-sight positions for which the correction has succeeded, and the estimated value is set as a new line-of-sight position before correction of the failed line-of-sight position.

[0008] A correction program according to one aspect of the present invention causes a computer to function as the above-described correction device.

Effect of the Invention

[0009] According to the present invention, it is possible to provide a technique capable of improving the detection accuracy of the line-of-sight position.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are denoted by the same reference numerals and the description thereof will be omitted.

[0012] [SUMMARY] To solve the above problems, the present disclosure corrects the distortion (shift of the line-of-sight position) of the line-of-sight position of the user detected by the computer. Thus, since the distortion of the line-of-sight position of the user is corrected, the detection accuracy of the line-of-sight position of the user can be improved.

[0013] [DEVICE CONFIGURATION] FIG. 1 is a diagram showing a configuration example of the correction device 1 and the tablet terminal 2 according to the present embodiment.

[0014] The correction device 1 functions inside the tablet terminal 2, acquires the line-of-sight position of the user U detected by the tablet terminal 2, and corrects the distortion of the line-of-sight position. That is, the correction device 1 corrects the line-of-sight position AT of the user U detected by the tablet terminal 2 when the user U gazes at the target point TG to a line-of-sight position AT' close to the target point TG.

[0015] The correction device 1 includes a correction unit 11 that corrects the distortion of the line-of-sight position of the user U, a determination unit 12 that determines whether the tablet terminal 2 has normally detected the line-of-sight position of the user U, and a notification unit 13 that notifies various information to the user U in characters or voice.

[0016] The tablet terminal 2 includes a camera 21 disposed at an edge of the monitor screen, a monitor screen 22 that displays an image, a speaker 23 that outputs various information in voice, a detection unit 24 that detects the line-of-sight position of the user U from the image of the user U, and a storage unit 25 that stores various information and various data.

[0017] Note that the tablet terminal 2 is an example of a computer (general-purpose computer) that can photograph the user U with a camera and detect the line-of-sight position of the user U from the image of the user U. Such a computer may be a smartphone terminal, a personal computer device, or the like.

[0018] Also, the correction device 1 may include, in its own device, a functional unit having the same function as the detection unit 24 of the tablet terminal 2. In this case, the correction device 1 can detect the line-of-sight position of the user U from the image of the user U by itself.

[0019] [Function of Correction Device 1] The correction unit 11 acquires each line-of-sight position AT of the user U detected by the tablet terminal 2 when the user U photographed by the camera 21 gazes at each of a plurality of target points TG sequentially displayed within the monitor screen 22, and has a function of correcting the distortion of the line-of-sight position AT of the user U based on the fact that the camera 21 photographed the user U from the corner direction (specific direction) of the edge of the monitor screen 22.

[0020] The correction unit 11 has a function of correcting the distortion of the line-of-sight position AT of the user U by using the line-of-sight position AT of the user U detected after a predetermined time (for example, 500 ms) has elapsed since the target point TG was displayed on the monitor screen 22.

[0021] The correction unit 11 determines whether or not the line-of-sight position of the user U remains within the threshold region of the monitor screen 22 continuously for a predetermined time (for example, 500 ms), and has a function of correcting the distortion of the line-of-sight position AT of the user U by using the line-of-sight position AT of the user U that remains within the threshold region.

[0022] The correction unit 11 generates a conversion formula for converting the distortion of the line-of-sight position AT before correction so that the distance between the plurality of target points TG and each corrected line-of-sight position AT' becomes minimum, and has a function of correcting the distortion of the line-of-sight position AT of the user U by using the conversion formula.

[0023] The correction unit 11 determines whether or not the correction of the distortion of the line-of-sight positions of the plurality of users U with respect to the plurality of target points TG has been successful based on a predetermined criterion, and if not all have been successful, has a function of automatically repeating the re-correction of the distortion of the line-of-sight positions AT of the plurality of users U.

[0024] The correction unit 11 has a function of repeating the re-correction of the distortion of the line-of-sight positions AT of the plurality of users U a predetermined number of times (for example, up to 9 times).

[0025] When the correction unit 11 determines success or failure of correction based on a predetermined criterion and there is a line-of-sight position where the correction has failed, the correction unit 11 calculates an estimated value of the line-of-sight position before correction related to the failed line-of-sight position using each line-of-sight position before correction of a plurality of line-of-sight positions where the correction has succeeded, and sets the estimated value as the new line-of-sight position before correction of the failed line-of-sight position.

[0026] For example, after the correction unit 11 repeats re-correction a predetermined number of times, if only one of the corrections of the distortion of the line-of-sight positions AT of a plurality of users U with respect to a plurality of target points TG fails, the correction unit 11 calculates an estimated value of the line-of-sight position before correction related to that one correction using each line-of-sight position before correction of the other plurality of line-of-sight positions where the correction has succeeded, and sets it as the new line-of-sight position before correction.

[0027] The determination unit 12 has a function of determining whether or not the tablet terminal 2 has normally detected the line-of-sight position AT of the user U before the correction unit 11 corrects the distortion of the line-of-sight position AT of the user U, or after the correction of the distortion of the line-of-sight position AT of the user U by the correction unit 11 fails. For example, the determination unit 12 has a function of determining whether or not mirroring has occurred in the line-of-sight position AT of the user U detected by the tablet terminal 2.

[0028] When mirroring has occurred in the line-of-sight position AT of the user U detected by the tablet terminal 2, the determination unit 12 has a function of causing the correction unit 11 to execute dummy correction. When mirroring has occurred, the correction unit 11 has a function of performing dummy correction that only causes the detection unit 24 of the tablet terminal 2 to detect the line-of-sight position of the user U.

[0029] The notification unit 13 has a function of notifying the user U of teaching information regarding the tilt of the user U's face or teaching information regarding the blink of the user U by voice or text.

[0030] Note that the target points are, for example, a plurality of target points displayed in a grid pattern at the center and periphery within the monitor screen 22. Five or more target points are preferable. For example, five in total at the four corners of the periphery including the center within the monitor screen 22, five in total at the midpoints of each of the four sides of the periphery including the center, nine in total at the four corners of the periphery and the midpoints of each of the four sides including the center, and ten including the center again among the nine are more preferable. The correction unit 11 corrects, for example, the distortion of five or more line-of-sight positions related to five or more target points. By setting the number of target points and the number of line-of-sight positions to be corrected to five or more, the detection accuracy of the user's line-of-sight position can be further improved.

[0031] Note that the above-described correction unit 11, determination unit 12, and notification unit 13 may be provided in one functional unit (for example, one processing unit).

[0032] [Implementation Example of Correction Device 1] The correction device 1 may be implemented outside the tablet terminal 2. For example, it can be implemented in a server device within the Internet or in the cloud that is communicably connected to the tablet terminal 2. Also, the correction device 1 can be provided as a software program (including an application-type software program) that causes the tablet terminal 2 to function as the correction device 1.

[0033] [Overall Operation] The operation when examining the eye movement function or the like of the subject U who is the user U will be described. The subject U places their chin on the chin rest installed in front of the tablet terminal 2 and faces their face toward the camera 21 and the monitor screen 22 of the tablet terminal 2.

[0034] Figure 2 is a diagram showing the overall processing flow of the correction device 1 and the tablet terminal 2.

[0035] Step S101; The subject U or the examiner activates the camera 21 of the tablet terminal 2.

[0036] Step S102; The detection unit 24 of the tablet terminal 2 detects the line-of-sight position of the subject U from the face image of the subject U captured by the camera 21. The line-of-sight position is the position of the fixation point of the subject U with respect to the monitor screen 22.

[0037] Step S103; The determination unit 12 of the correction device 1 acquires the position of the eyes and the line-of-sight position of the subject U from the detection unit 24 of the tablet terminal 2, and determines whether the detection unit 24 has normally detected the line-of-sight position of the subject U. For example, the determination unit 12 determines the presence or absence of mirroring. If the detection unit 24 has not normally detected the line-of-sight position of the subject U, the process returns to step S101 and step S102 is automatically repeated.

[0038] Step S104; When the detection unit 24 has normally detected the line-of-sight position of the subject U, the correction unit 11 of the correction device 1 corrects the distortion of the line-of-sight position of the subject U. The distortion of the line-of-sight position is the deviation of the line-of-sight position based on the fact that the camera 21 has photographed the subject U from the corner of the monitor screen 22.

[0039] Step S105; The correction unit 11 of the correction device 1 determines whether the correction of the distortion of the line-of-sight position of the subject U has been successful based on a predetermined standard. When the correction of the distortion of the line-of-sight position of the subject U has been successful, the correction unit 11 outputs the corrected line-of-sight position to an inspection device such as an eye movement function or the inspection unit in the tablet terminal 2, and ends the process. If the correction of the distortion of the line-of-sight position of the subject U has failed, the process returns to step S103 and is automatically repeated.

[0040] In this way, since the correction device 1 corrects the distortion of the line-of-sight position of the subject U detected by the tablet terminal 2, the detection accuracy of the line-of-sight position of the subject U can be improved. The corrected line-of-sight position is used for the inspection of the eye movement function and other functions of the subject U, which is the purpose.

[0041] Such a correction device 1 is applicable to a diagnostic medical device that indicates the possibility of schizophrenia and other mental disorders based on various inspection results such as eye movement function and cognitive function. In addition, the correction device 1 is also applicable to fields other than the medical field.

[0042] Note that steps S104 and S105 may be executed before step S103. In this case, if the correction of the distortion of the line-of-sight position is successful in one attempt in step S104, there is no need to perform step S103, so the correction processing time can be shortened.

[0043] [Correction Process for Distortion of Line-of-Sight Position] Next, the correction process for the distortion of the line-of-sight position performed in step S104 of FIG. 2 will be described.

[0044] FIG. 3 is a diagram showing the correction process flow for the distortion of the line-of-sight position.

[0045] Steps S201, S202; The correction unit 11 of the correction device 1 executes calibration. If the first calibration fails, the correction unit 11 executes the second calibration.

[0046] Specifically, the correction unit 11 displays a plurality of target points (for example, 10 points) on the monitor screen 22 of the tablet terminal 2, and for a plurality of target points (for example, 9 points) excluding the target points at the same position, derives a conversion formula for correcting the distortion of each line-of-sight position of the subject U detected by the detection unit 24 of the tablet terminal 2 to the correct line-of-sight position, and evaluates the conversion formula.

[0047] Steps S203, S204; If the calibration is successful in step S201 or step S202, the correction unit 11 of the correction device 1 executes validation. If the first validation fails, the correction unit 11 executes the second validation.

[0048] Specifically, the correction unit 11 displays a plurality of target points on the monitor screen 22 of the tablet terminal 2 in the same manner as calibration, corrects the distortion of each line-of-sight position of the subject U using the conversion formula derived during calibration, and determines that the correction is successful if the distances between all of the target points are equal to or less than the threshold value.

[0049] Step S205; If the validation is successful in step S203 or step S204, that is, if the correction of the distortion of each line-of-sight position for the nine target points is all successful, the correction unit 11 of the correction device 1 determines whether the elapsed time from the start of the first calibration to the present exceeds a threshold value (for example, 70 seconds). If the elapsed time to the present does not exceed the threshold value, the correction unit 11 normally ends the process.

[0050] Step S206; If the calibration fails in step S202, if the validation fails in step S204 (if the correction is not successful for even one point), or if the elapsed time to the present exceeds the threshold value in step S205, the correction unit 11 of the correction device 1 determines the total number of retries for the entire inspection.

[0051] Step S207; If the total number of retries for the entire inspection is between 1 and 8, the notification unit 13 of the correction device 1 outputs an error message indicating that the correction has failed to the monitor screen 22 of the tablet terminal 2, or outputs the error message as audio from the speaker 23. Then, it returns to step S201. Exactly, it returns to step S103 in FIG. 2 and resumes the process of determining whether the detection unit 24 has normally detected the line-of-sight position of the subject U.

[0052] Step S208; If the total number of retries for the entire inspection is 9, the correction unit 11 of the correction device 1 determines whether there is at least one instance where the correction of 8 out of 9 points has been successful. If there is not even one instance where the correction of 8 points has been successful, the correction unit 11 abnormally ends the inspection.

[0053] Step S209; When the correction unit 11 of the correction device 1 has had one or more successful corrections for eight points, the correction unit 11 estimates the line-of-sight position before correction for the one point that had a correction failure (one-point correction), generates a conversion formula for correcting the distortion of the line-of-sight position based on the line-of-sight position before correction, and corrects the distortion of the line-of-sight position using the conversion formula. After that, the correction unit 11 normally ends the process.

[0054] [Calibration] Next, the calibration performed in steps S201 and S202 of FIG. 3 will be described.

[0055] FIG. 4 is a diagram showing the processing flow of calibration.

[0056] Step S301; The correction device 1 causes the tablet terminal 2 to display ten target points one by one on the monitor screen 22 of the tablet terminal 2.

[0057] For example, as shown in FIG. 5, the correction device 1 displays ten target points one by one in a grid pattern at the center and the periphery within the monitor screen 22. The first target point TG1 is displayed at the center, and after deleting the first target point TG1, the second target point TG2 is displayed at the upper left.

[0058] Similarly, the third to ninth target points TG3 to TG9 are displayed in that order from the upper right → lower right → lower left → upper center → right center → lower center → left center. Finally, the tenth target point TG10 is displayed at the center.

[0059] That is, the correction device 1 first displays the central point, then displays the peripheral points, and finally redisplays the central point.

[0060] Note that the display order of the target points may be random. For example, the first target point may be displayed at the center, the second to ninth target points may be displayed in a random order, and the tenth target point may be redisplayed at the center. Alternatively, the first target point may be displayed at the upper left and the tenth target point may be displayed at the upper right.

[0061] Step S302; At this time, the subject U is gazing at each of the target points TG1 to TG10 in order. The detection unit 24 of the tablet terminal 2 detects the line-of-sight position of the subject U. Thereafter, the correction unit 11 of the correction device 1 acquires the coordinates of the line-of-sight position of the subject U (the coordinates of the fixation point) from the detection unit 24.

[0062] Here, the method for acquiring the coordinates of the line-of-sight position will be described.

[0063] As shown in FIG. 6, after a certain period (for example, 500 ms) has elapsed since the nth (n is a natural number) target point is displayed, the detection unit 24 of the tablet terminal 2 starts detecting the line-of-sight position.

[0064] Then, the correction unit 11 of the correction device 1 acquires the xy coordinate group of the line-of-sight position of the subject U (the xy coordinate group of the fixation point) detected within a certain period (for example, 500 ms) after the subject U starts gazing at the nth target point.

[0065] Thereafter, the correction unit 11 performs a fixation determination as to whether or not the acquired xy coordinate group of the line-of-sight position falls within a preset threshold value. For example, as shown in FIG. 7, after 500 ms has elapsed since the nth target point is displayed, for the xy coordinate group of the line-of-sight position detected within the 500-ms fixation period, a rectangle circumscribing the xy coordinate group of the line-of-sight position is calculated, and it is determined whether or not the diagonal d of the rectangle is less than or equal to the threshold value.

[0066] After 500 ms has elapsed since the target point is displayed, every 100 ms, fixation determination is respectively performed on the xy coordinate group of the line-of-sight positions within the fixation period every 500 ms, and the process is repeatedly continued until the fixation determination is successful. If the diagonal d of the rectangle is less than or equal to the threshold value, the fixation determination is successful.

[0067] Finally, the correction unit 11 of the correction device 1 calculates the median value of the xy coordinate group of the line-of-sight positions within the fixation period in which the fixation determination is successful, and sets the median value as the xy coordinates of the line-of-sight position of the subject U.

[0068] As a result, for the 10 target points, the xy coordinates (median values) of the 10 line-of-sight positions are acquired. However, since the positions of the first and tenth target points are the same, the xy coordinates (median values) of the 9 line-of-sight positions related to the 9 target points from the second to the tenth are used.

[0069] Step S303; The line-of-sight position acquired in step S302 is the central point of the xy coordinate group of the line-of-sight positions within the fixation period, but it is the line-of-sight position detected by the tablet terminal 2, and there is distortion in the line-of-sight position based on the fact that the camera 21 photographed the subject U from the corner direction at the edge of the monitor screen 22.

[0070] Specifically, the direction of the camera 21 (shooting direction) and the direction of the subject U (line-of-sight direction) are not on the same axis, and the lens size of the camera 21 is much smaller than the face size of the subject U, so distortion occurs in the line-of-sight position of the subject U detected by the tablet terminal 2. Also, the distortion degree is larger for the line-of-sight position at a position farther from the camera 21 than for the line-of-sight position at a position closer to the camera 21.

[0071] Therefore, in calibration and validation, this distortion of the line-of-sight position of the subject U is corrected by polynomial fitting. In calibration, a conversion formula as the polynomial is derived.

[0072] The correction unit 11 of the correction device 1 derives a conversion formula for converting the xy coordinates of the nine target points displayed in step S301 and the xy coordinates of the gaze position of the subject U after conversion (after correction) so that the distance therebetween becomes minimum, from the xy coordinates (median values) of the gaze positions of the nine subject Us obtained in step S302.

[0073] FIG. 8 is a diagram showing a derivation flow of the conversion formula.

[0074] Step S303(1); For each of the nine target points, the correction unit 11 calculates the median values (x, y) of the xy coordinate groups of the gaze positions within the fixation period in which the fixation determination has succeeded. The calculation of the median value has already been described, but is described again here. At this time, the correction unit 11 also calculates the center point (x_center, y_center) of the monitor screen 22 of the tablet terminal 2. Note that the center point may use the coordinate values of the tenth target point.

[0075] Step S303(2); The correction unit 11 converts each of the median values (x, y) (raw coordinates) of the nine gaze positions into a coordinate system with the center of the monitor screen 22 of the tablet terminal 2 as the origin. Specifically, the center point (x_center, y_center) is subtracted from each of the nine raw coordinates (x, y).

[0076] Step S303(3); The correction unit 11 defines, for example, the following conversion formulas (1) and (2), where the xy coordinates of the gaze position of the subject U before conversion (raw coordinates after coordinate system conversion) are x and y, and the xy coordinates of the gaze position of the subject U after conversion are X and Y, and calculates them by the least squares method.

[0077] X = a00×x + a01×y + a02×x×y + a03×x 2 + a04×y 2 + a05×(x×y) 2 + a06 …(1) Y = a10×x + a11×y + a12×x×y + a13×x 2 + a14×y 2 + a15×(x×y)2 +a16 …(2) The coefficients a00 to a06 and a10 to a16 of the conversion formulas (1) and (2) are determined by the least squares method so that the conversion of the line-of-sight position is optimized (that is, the xy coordinates after conversion are closest to the xy coordinates of the target point). Fitting is performed with two-dimensional polynomials for x and y respectively.

[0078] Step S304; The correction unit 11 evaluates the conversion formula derived in step S303.

[0079] Specifically, the correction unit 11 substitutes the coordinates (x, y) of the nine pre-conversion line-of-sight positions obtained in step S302 into the conversion formula derived in step S303 to calculate the coordinates (X, Y) of the nine post-conversion line-of-sight positions, and calculates the distances from the coordinates of the nine target points (the errors from the coordinates of the line-of-sight positions) respectively.

[0080] The correction unit 11 determines Good if the distance is less than or equal to 1 / 2 of the threshold for Fair determination, determines Fair if the distance is less than or equal to the threshold for Fair determination, and determines Poor if the distance exceeds the threshold for Fair determination.

[0081] Then, if the determination results of the errors for all nine points are within Fair (Good or Fair), the correction unit 11 determines that the conversion formula is OK and determines that the calibration is successful. If there is even one Poor, the correction unit 11 determines that the conversion formula is NG and determines that re-detection and re-correction of all nine points are necessary.

[0082] [Validation] Next, the validation performed in steps S203 and S204 of FIG. 3 will be described.

[0083] FIG. 9 is a diagram showing the processing flow of validation.

[0084] Step S401; It is the same as step S301 in FIG. 4.

[0085] Step S402; It is the same as step S302 in FIG. 4.

[0086] Step S403; The correction unit 11 of the correction device 1 converts the coordinates of the gaze position of the subject U obtained in step S402 using the conversion formula determined to be successful in calibration in step S304 of FIG. 4.

[0087] Step S404; Similar to step S304, the correction unit 11 of the correction device 1 calculates the distance from the coordinates of the nine target points to the coordinates of the converted gaze position of the subject U (the error from the coordinates of the gaze position), and determines the success or failure of validation based on the degree of the length of the distance (the accuracy of the error).

[0088] FIG. 10 is a diagram showing an example of validation state transition.

[0089] In the validation state V0, the distance between the coordinates of the target point and the coordinates of the converted gaze position is calculated for each gaze position. If it is within Fair (Good or Fair) for all nine target points, it is determined that the validation is successful and the transition is to OK. If there is even one remaining target point with Poor, it is targeted for retry and the transition is to the validation state V1.

[0090] In the validation state V1, only the target points targeted for retry are redisplayed, the coordinates of the converted gaze position are recalculated, the distance between the coordinates of the target point and the coordinates of the converted gaze position is calculated, and if it is within Fair (Good or Fair), it is determined that the validation is successful and the transition is to OK.

[0091] On the other hand, if there is even one remaining target point for which the validation is not successful after retry, it is determined that the validation has failed and the transition is to NG. At this time, it is determined that redetection and recorrection of all nine points are required.

[0092] Note that when retrying, each target point may be displayed in a random order and position. When retrying multiple times for only the remaining one point, it is turned off before the next display and then turned on.

[0093] [Method of single-point correction] Next, the single-point correction performed in step S209 of FIG. 3 will be described.

[0094] In step S204 of FIG. 3, if there is even one remaining line-of-sight position that failed validation, the process transitions to step S206. And when the total number of retries for the entire inspection is small, after notifying an error message in step S207, the process returns to step S201 (accurately, returns to step S103 shown in FIG. 2), and the correction of the distortion of all nine line-of-sight positions is re-executed.

[0095] On the other hand, when the total number of retries for the entire inspection increases, even if the correction of the distortion of the line-of-sight position is repeated, it is considered difficult to correct the distortion of all nine line-of-sight positions. At this time, the correction of the distortion of the line-of-sight position may be immediately terminated, but if the correction has been successful for many of the nine points, it is preferable to utilize the data at that time.

[0096] Therefore, when the total number of retries for the entire inspection is nine, even if calibration and validation have failed, if there is a case where only one point does not meet the standard, the line-of-sight position of that one point that does not meet the standard is estimated from other points, a conversion formula is generated, and the distortion of the line-of-sight position is corrected using that conversion formula.

[0097] For example, as shown in FIG. 11, assume that the corrected line-of-sight positions for the nine target points TG2 to TG10 are AT2' to AT10', and only AT3' is determined to be Poor.

[0098] In this case, as shown in the upper part of FIG. 12, the correction unit 11 of the correction device 1 deletes the line-of-sight position AT3 before correction that was determined to be Poor.

[0099] Then, as shown in the lower part of FIG. 12, the correction unit 11 of the correction device 1 draws a straight line L1 passing through two line-of-sight positions AT2 and AT6 before conversion (before correction) in the same row, draws a straight line L2 passing through two line-of-sight positions AT7 and AT4 before conversion in the same column, and sets the intersection of the two straight lines L1 and L2 as a new line-of-sight position AT3 before conversion.

[0100] After that, the correction unit 11 of the correction device 1 derives a conversion formula in the same manner as in step S303 of FIG. 4, and corrects the distortion of the line-of-sight positions AT2 to AT10 before correction with the conversion formula in the same manner as in step S403 of FIG. 9.

[0101] Note that the correction unit 11 of the correction device 1 may calculate a new line-of-sight position AT3 before conversion using a straight line (L3) passing through two other line-of-sight positions AT5 and AT10 instead of the straight line L1 or the straight line L2. Additionally, the correction unit 11 may select one of the intersection of the straight line L1 and the straight line L2, the intersection of the straight line L1 and the straight line L3, and the intersection of the straight line L2 and the straight line L3.

[0102] Note that when there are multiple cases where only one point out of the nine retry counts does not meet the standard, the data of the latest time is used.

[0103] So far, the case where one-point correction fails has been described as an example. However, it is applicable not only when one-point correction fails, but also when two-point correction or three-point correction fails. That is, when there is a line-of-sight position where correction fails, an estimated value of the line-of-sight position before correction related to the failed line-of-sight position is calculated using each line-of-sight position before correction of a plurality of line-of-sight positions where correction is successful, and the estimated value is set as a new line-of-sight position before correction of the failed line-of-sight position.

[0104] [Notification during calibration or validation] The notification unit 13 of the correction device 1, according to the progress of the inspection before, during, and after the start of calibration or validation, based on the detection result of the face image of the subject U detected by the tablet terminal 2, displays the following teaching information in characters on the monitor screen 22 of the tablet terminal 2, and outputs the teaching information as voice from the speaker 23 of the tablet terminal 2.

[0105] For example, the notification unit 13 outputs instruction information such as "Then, start adjusting the machine. Please look at the points that appear on the screen. Without moving your head, look at the points that appear one after another. When looking at the points, please stop blinking for a while and keep looking until the points disappear. Then, let's start. Please look at the point in the middle."

[0106] For example, the notification unit 13 outputs instruction information such as "Do it again. Please look at the point in the middle. Then, points will appear at the edges of the screen, so please keep looking steadily without closing your eyes until the points disappear. Then, let's start. Please look at the point in the middle."

[0107] For example, the notification unit 13 outputs instruction information such as "Do it again. Please look at the points that appear on the screen. Without moving your head, look at the points that appear one after another. When looking at the points, please stop blinking for a while, keep your eyes open without closing them, and keep looking steadily until the points disappear. Then, let's start. Please look at the point in the middle."

[0108] For example, the notification unit 13 outputs instruction information such as "It seems that you are not looking firmly at the middle of the point. While the point is out, please keep your eyes wide open and look steadily until the point disappears. Relax once, then put your hand on the table again, rest your chin on the chin rest as if peering into the screen, and take a comfortable posture. Please refer to the figure on the right for the angle with the screen."

[0109] [Determination Process for Normal Detection of Line-of-Sight Position] Next, the determination process for normal detection of the line-of-sight position performed in step S103 of FIG. 2 will be described.

[0110] There may be a case where the line-of-sight position of the subject U cannot be detected normally by the tablet terminal 2 for some reason. In this case, steps S104 and S105 in FIG. 2 cannot be executed appropriately, and subsequent eye movement function tests and the like cannot proceed normally. Therefore, when any abnormality occurs, the correction device 1 interrupts the test once using a certain criterion and performs a retest.

[0111] In this embodiment, as an example of any abnormality, a case where mirroring occurs in the tablet terminal 2 will be described. When mirroring is detected before or after the start of steps S104 and S105 in FIG. 2, the test for that time is interrupted as if it had not been performed. After taking a break once, steps S104 and S105 of the same time are performed again.

[0112] And when mirroring occurs continuously for the number of consecutive occurrences threshold (for example, 3 times), the test ends abnormally. Even if mirroring has not occurred continuously for the threshold number of times, if mirroring has occurred for the cumulative number of occurrences threshold (for example, cumulative 3 times) throughout the test, the test ends abnormally.

[0113] Here, mirroring will be described.

[0114] As shown in FIG. 13, the tablet terminal 2 originally detects the subject U at a position on the front side of the tablet terminal 2. Specifically, it detects at a position on the “−” side on the z-axis, which is the depth direction of the monitor screen 22. However, there may be a case where the subject U is detected as the subject U' at a position on the “+” side on the z-axis. This phenomenon is called mirroring.

[0115] When such mirroring occurs, the position of the subject U will be misrecognized, so it is difficult to appropriately handle the line-of-sight position of the subject U. Therefore, by changing the position of the head or the like and restarting the camera 21 of the tablet terminal 2, the mirroring is eliminated.

[0116] FIG. 14 is a diagram showing a determination processing flow of the presence or absence of mirroring.

[0117] Step S501; The subject U or the examiner activates the camera 21 of the tablet terminal 2. If the camera 21 has already been activated, step S501 is omitted.

[0118] Step S502; The determination unit 12 of the correction device 1 determines whether mirroring has occurred on the tablet terminal 2. For example, after the activation of the camera 21, the determination unit 12 determines whether the value of the position of the eyes or head on the z-axis detected by the detection unit 24 of the tablet terminal 2 is greater than '0'.

[0119] Step S503; If the value of the position of the eyes or head on the z-axis is greater than '0', the determination unit 12 of the correction device 1 determines that mirroring has occurred and determines the number of consecutive occurrences of mirroring.

[0120] Step S504; If the number of consecutive occurrences of mirroring is 1, the determination unit 12 of the correction device 1 determines the cumulative number of occurrences of mirroring in the entire inspection.

[0121] Step S505; If the number of consecutive occurrences of mirroring is 2 and the cumulative number of occurrences of mirroring in the entire inspection is 1, the determination unit 12 of the correction device 1 instructs the correction unit 11 to acquire the gaze position of the subject U and to correct (dummy correction) the distortion of the gaze position of the subject U. At this time, the correction unit 11 corrects (dummy corrects) the distortion of the gaze position of the subject U. Dummy correction will be described later.

[0122] At this time, the notification unit 13 of the correction device 1 notifies, for example, the following teaching information in characters or voice: "Please look at the points that appear on the screen. Please keep your head still and look at the points that appear one after another. When looking at the points, please blink slightly less often and keep looking until the points disappear. Then let's start. Please look at the point in the middle."

[0123] Step S506; After step S505, the notification unit 13 of the correction device 1 instructs the subject U to change the position of the head in characters or voice in order to cancel the mirroring.

[0124] For example, the notification unit 13 notifies the instructional information such as "Your face may have moved. Please relax once, place your hand on the table again, rest your chin on the chin rest as if looking closely at the screen, and take a comfortable posture. Please refer to the figure on the right for the angle with the screen." in characters or voice.

[0125] After that, it returns to step S501. Exactly, it returns to step S101 in FIG. 2. Then, the camera is restarted in step S501, and it is re-determined whether mirroring has occurred in step S502.

[0126] Step S507; When the number of consecutive occurrences of mirroring is 3 or when the cumulative number of occurrences of mirroring in the entire inspection is 2, the determination unit 12 of the correction device 1 corrects (dummy correction) the distortion of the subject U's line-of-sight position in the same way as in step S505.

[0127] Step S508; After step S507, the determination unit 12 of the correction device 1 determines that it is difficult to cancel the mirroring on the tablet terminal 2 and ends the inspection abnormally.

[0128] Note that the dummy correction performed in step S505 or step S507 is actually a measurement of the line-of-sight position that only appears to obtain the desired data, that is, a process of only causing the detection unit 24 of the tablet terminal 2 to detect the line-of-sight position of the subject U.

[0129] Since the mirroring determination ends immediately, it may give the impression to the subject U that the inspection of the eye movement function, etc. has suddenly ended. However, by performing the dummy correction in step S505 and step S507, the mental burden of the subject U on the inspection can be reduced. However, in the dummy correction, it is also possible to actively perform the distortion correction of the subject U's line-of-sight position using the detected line-of-sight position of the subject U.

[0130] Step S509; When the value of the position of the eyes or head on the z-axis is '0' or less in step S502, the determination unit 12 of the correction device 1 determines that no mirror imaging has occurred and ends the process normally.

[0131] Note that the above-mentioned number of consecutive occurrences and cumulative number of occurrences of mirror imaging are examples. For example, the number of consecutive occurrences may be set to 5, or the cumulative number of occurrences may be set to 7. Also, the determination of the presence or absence of mirror imaging is performed, for example, within a time period of 0.5 seconds.

[0132] [Effect] According to the present embodiment, the correction unit 11 acquires each line-of-sight position AT of the user U detected by the tablet terminal 2 when the user U photographed by the camera 21 gazes at each of a plurality of target points TG sequentially displayed within the monitor screen 22, and corrects the distortion of the line-of-sight position AT of the user U based on the fact that the camera 21 photographs the user U from the corner direction of the edge of the monitor screen 22. Therefore, a technique capable of improving the detection accuracy of the line-of-sight position can be provided.

[0133] According to the present embodiment, the correction unit 11 determines success or failure of correction based on a predetermined criterion. When there is a line-of-sight position where correction has failed, the correction unit 11 uses each line-of-sight position AT before correction of a plurality of line-of-sight positions where correction has succeeded to calculate an estimated value of the line-of-sight position AT before correction related to the failed line-of-sight position, and sets the estimated value as the new line-of-sight position AT before correction of the failed line-of-sight position. Therefore, a technique capable of further improving the detection accuracy of the line-of-sight position can be provided.

[0134] According to the present embodiment, the correction unit 11 corrects the distortion of the line-of-sight position AT of the user U using the line-of-sight position AT of the user U detected after 500 ms from when the target point TG is displayed on the monitor screen 22. Therefore, a technique capable of further improving the detection accuracy of the line-of-sight position can be provided.

[0135] According to the present embodiment, the correction unit 11 determines whether or not the user U's line-of-sight position remains within the threshold region of the monitor screen 22 continuously for 500 ms, and corrects the distortion of the user U's line-of-sight position AT using the user U's line-of-sight position AT that falls within the threshold region. Therefore, a technique capable of further improving the detection accuracy of the line-of-sight position can be provided.

[0136] According to the present embodiment, the correction unit 11 generates a conversion formula for converting the distortion of the line-of-sight position AT before correction so that the distance between the plurality of target points TG and each corrected line-of-sight position AT' becomes minimum, and corrects the distortion of the user U's line-of-sight position AT using the conversion formula. Therefore, a technique capable of further improving the detection accuracy of the line-of-sight position can be provided.

[0137] According to the present embodiment, the correction unit 11 corrects the distortion of the user's line-of-sight position by five or more, so that a technique capable of further improving the detection accuracy of the line-of-sight position can be provided.

[0138] According to the present embodiment, the plurality of target points are five or more target points displayed in a grid pattern at the center and the periphery within the screen, the central point is displayed, and then the peripheral points are displayed. Therefore, a technique capable of further improving the detection accuracy of the line-of-sight position can be provided.

[0139] According to the present embodiment, the determination unit 12 determines whether or not the tablet terminal 2 has normally detected the user U's line-of-sight position AT before the correction unit 11 corrects the distortion of the user U's line-of-sight position AT or after the correction of the distortion of the user U's line-of-sight position AT by the correction unit 11 fails. Therefore, a technique capable of further improving the detection accuracy of the line-of-sight position can be provided.

[0140] According to the present embodiment, when mirroring occurs in the user U's line-of-sight position detected by the tablet terminal 2, the correction unit 11 performs dummy correction that only causes the tablet terminal 2 to detect the user U's line-of-sight position. Therefore, the mental load on the subject U for the examination can be reduced.

[0141] According to this embodiment, since the notification unit 13 notifies the user U of teaching information regarding the tilt of the face of the user U or teaching information regarding the blinking of the user U, it is possible to provide a technology capable of further improving the detection accuracy of the line-of-sight position.

[0142] [Others] The present invention is not limited to the above-described embodiment. The present invention can be variously modified within the scope of the gist of the present invention.

[0143] The correction device 1 of the above-described embodiment can be realized by using, for example, a general-purpose computer system including a CPU 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906 as shown in FIG. 15. The memory 902 and the storage 903 are storage devices. In the computer system, each function of the correction device 1 is realized by the CPU 901 executing a predetermined program loaded onto the memory 902.

[0144] The correction device 1 may be implemented by one computer. The correction device 1 may be implemented by a plurality of computers. The correction device 1 may be a virtual machine implemented on a computer. The program for the correction device 1 can be stored in a computer-readable recording medium such as an HDD, an SSD, a USB memory, a CD, or a DVD. A computer-readable recording medium is, for example, a non-transitory recording medium. The program for the correction device 1 can also be distributed via a communication network.

Explanation of Reference Numerals

[0145] 1 Correction device 11 Correction unit 12 Determination unit 13 Notification unit 2 Tablet terminal 21 Camera 22 Monitor screen 23 Speaker 24 Detection unit 25 Storage unit 901…CPU 902…Memory 903… Storage 904… Communication device 905… Input device 906… Output device

Claims

1. Obtaining each line-of-sight position of the user detected by a computer when the user photographed by a camera gazes at a plurality of target points sequentially displayed within a screen, and comprising a correction unit that corrects distortion of the line-of-sight position of the user based on the fact that the camera photographed the user from a specific direction, The correction unit: Determines success or failure of the correction based on a predetermined criterion. When there is a line-of-sight position for which the correction has failed, using each line-of-sight position before correction of the plurality of line-of-sight positions for which the correction has succeeded, calculates an estimated value of the line-of-sight position before correction related to the failed line-of-sight position, and sets the estimated value as the new line-of-sight position before correction of the failed line-of-sight position. A correction device.

2. The correction unit: Generates a conversion formula that converts distortion of the line-of-sight position before correction so that the distance between the plurality of target points and each line-of-sight position after correction becomes minimum, and corrects the distortion of the line-of-sight position of the user using the conversion formula. The correction device according to claim 1.

3. The correction unit: Determines whether or not the line-of-sight position of the user continuously falls within a threshold region of the screen for a predetermined time, and corrects the distortion of the line-of-sight position of the user using the line-of-sight position of the user that falls within the threshold region. The correction device according to claim 1.

4. The correction unit: Corrects distortion of the line-of-sight position of the user by five or more. The correction device according to claim 1.

5. The plurality of target points: Are five or more target points displayed in a grid pattern at the center and the periphery within the screen, with the central point being displayed first and then the peripheral points being displayed. The correction device according to claim 1.

6. Before correcting the distortion of the line-of-sight position of the user, or after the correction of the distortion of the line-of-sight position of the user has failed, comprising a determination unit that determines whether or not the computer has normally detected the line-of-sight position of the user, The determination unit: Determines whether or not mirroring has occurred with respect to the line-of-sight position of the user detected by the computer. The correction device according to claim 1.

7. The correction unit: When mirroring has occurred with respect to the line-of-sight position of the user detected by the computer, performs dummy correction that only causes the computer to detect the line-of-sight position of the user. The correction device according to claim 6.

8. A notification unit that notifies the user of teaching information regarding the tilt of the user's face or teaching information regarding the user's blinking, The correction device according to claim 1, further comprising.

9. In a correction method performed by a correction device, When a user photographed by a camera gazes at a plurality of target points sequentially displayed on a screen, obtain each line-of-sight position of the user detected by a computer, and correct the distortion of the line-of-sight position of the user based on the fact that the camera photographed the user from a specific direction. Determine the success or failure of the correction based on a predetermined criterion. If there is a line-of-sight position where the correction fails, use each line-of-sight position before correction of the plurality of line-of-sight positions where the correction is successful to calculate an estimated value of the line-of-sight position before correction related to the failed line-of-sight position, and use the estimated value as the new line-of-sight position before correction of the failed line-of-sight position. Correction method.

10. A correction program that causes a computer to function as the correction device according to any one of Claims 1 to 8.

Images