Correction device, correction method, and correction program

Abstract

A technique capable of improving the detection accuracy of the gaze position is provided. [Solution] A correction unit 11 of a correction device 1 acquires each gaze position AT of a user U detected by a tablet terminal 2 when the user U photographed by a camera 21 of the tablet terminal 2 gazes at each of a plurality of target points TG displayed sequentially on a monitor screen 22, and corrects distortion (shift in gaze position) of the gaze position AT of the user U based on the camera 21 photographing the user U from the direction of an edge of the monitor screen 22. The correction unit 11 determines whether the correction is successful or unsuccessful based on a predetermined criterion, and if there is a gaze position for which the correction has failed, calculates an estimate of the gaze position AT before correction for the failed gaze position using each gaze position AT before correction for the multiple gaze positions for which the correction was successful, and sets the estimate value as a new gaze position AT before correction for the failed gaze position.

Description

[Technical field]

[0001] The present invention relates to a correction apparatus, a correction method, and a correction program. [Background technology]

[0002] Non-Patent Document 1 discloses a technology for testing a subject's cognitive function using a computer based on eye movements captured by a camera. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Evaluating cognitive function by analyzing eye movements in just 3 minutes: a new technology for early diagnosis of dementia," Osaka University, [online], [searched September 27, 2023],<URL: https: / / resou.osaka-u.ac.jp / ja / research / 2019 / 20190910_1> Summary of the Invention [Problem to be solved by the invention]

[0004] However, as shown in FIG. 1 of Non-Patent Document 1, the camera is placed at the edge of the monitor screen, which causes distortion in the subject's gaze position detected by the computer.

[0005] The present invention has been made in consideration of the above circumstances, and has an object to provide a technique capable of improving the detection accuracy of the gaze position. [Means for solving the problem]

[0006] A correction device of one embodiment of the present invention includes a correction unit that acquires each gaze position of a user detected by a computer when the user, photographed by a camera, gazes at each of a number of target points displayed sequentially on a screen, and corrects distortion of the user's gaze position based on the fact that the camera photographed the user from a specific direction, and the correction unit determines whether the correction is successful or unsuccessful based on a predetermined criterion.If there is a gaze position for which the correction has failed, an estimate of the pre-correction gaze position for the failed gaze position is calculated using each of the pre-correction gaze positions of the multiple gaze positions for which the correction was successful, and sets the estimate as the new pre-correction gaze position for the failed gaze position.

[0007] A correction method of one embodiment of the present invention is a correction method performed by a correction device, which obtains each gaze position of a user detected by a computer when a user photographed by a camera gazes at each of a number of target points displayed sequentially on a screen, corrects distortion of the user's gaze position based on the fact that the camera photographed the user from a specific direction, determines the success or failure of the correction based on a predetermined criterion, and if there is a gaze position for which the correction failed, calculates an estimate of the pre-correction gaze position for the failed gaze position using each of the pre-correction gaze positions of the multiple gaze positions for which the correction was successful, and sets the estimate as a new pre-correction gaze position for the failed gaze position.

[0008] A correction program according to an aspect of the present invention causes a computer to function as the correction device. Effect of the Invention

[0009] According to the present invention, it is possible to provide a technique capable of improving the detection accuracy of the gaze position. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a correction device and a tablet terminal. [Diagram 2] FIG. 2 is a diagram showing an overall processing flow of the correction device and the tablet terminal. [Diagram 3]FIG. 3 is a diagram showing a flow of processing for correcting distortion of the gaze position. [Figure 4] FIG. 4 is a diagram showing a processing flow of the calibration. [Diagram 5] FIG. 5 is a diagram showing an example of how the target points are displayed. [Figure 6] FIG. 6 is a diagram showing a method for acquiring the coordinates of the gaze position. [Figure 7] FIG. 7 is a diagram showing an image of the gaze determination method. [Figure 8] FIG. 8 is a diagram showing a flow of deriving the conversion formula. [Figure 9] FIG. 9 is a diagram showing a process flow of validation. [Figure 10] FIG. 10 is a diagram showing validation state transitions. [Figure 11] FIG. 11 is a diagram showing an example of correction of the distortion of the gaze position. [Figure 12] FIG. 12 is a diagram showing an image of one-point correction. [Figure 13] FIG. 13 is a diagram showing an image of mirror imaging. [Figure 14] FIG. 14 is a diagram showing a process flow for determining whether or not mirroring is performed. [Figure 15] FIG. 15 is a diagram illustrating an example of a hardware configuration of the correction device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[0012] [overview] In order to solve the above problem, the present disclosure corrects the distortion (deviation of gaze position) of a user's gaze position detected by a computer. In this way, the distortion of the user's gaze position is corrected, so that the detection accuracy of the user's gaze position can be improved.

[0013] [Device configuration] FIG. 1 is a diagram showing an example of the configuration of a correction device 1 and a tablet terminal 2 according to this embodiment.

[0014] The correction device 1 is a device that functions inside the tablet terminal 2, acquires the gaze position of the user U detected by the tablet terminal 2, and corrects distortion of the gaze position. In other words, the correction device 1 corrects the gaze position AT of the user U detected by the tablet terminal 2 when the user U gazes at the target point TG to a gaze position AT' close to the target point TG.

[0015] The correction device 1 includes a correction unit 11 that corrects distortion in the gaze position of the user U, a judgment unit 12 that judges whether the tablet terminal 2 has correctly detected the gaze position of the user U, and a notification unit 13 that notifies the user U of various information by text or voice.

[0016] The tablet terminal 2 comprises a camera 21 positioned at the edge of the monitor screen, a monitor screen 22 that displays images, a speaker 23 that outputs various information as audio, a detection unit 24 that detects the gaze position of user U from an image of user U, and a memory unit 25 that stores various information and data.

[0017] The tablet terminal 2 is an example of a computer (general-purpose computer) that can capture an image of the user U with a camera and detect the gaze position of the user U from the image of the user U. Such a computer may be a smartphone terminal, a personal computer device, or the like.

[0018] Furthermore, the correction device 1 may itself include a functional unit that is the same as the detection unit 24 of the tablet terminal 2. In this case, the correction device 1 can detect the gaze position of the user U from an image of the user U by the correction device 1 itself.

[0019] [Function of correction device 1] The correction unit 11 has a function of acquiring each gaze position AT of the user U detected by the tablet terminal 2 when the user U photographed by the camera 21 gazes at each of the multiple target points TG displayed sequentially on the monitor screen 22, and correcting distortion of the gaze position AT of the user U based on the fact that the camera 21 photographed the user U from the direction of the edge of the monitor screen 22 (a specific direction).

[0020] The correction unit 11 has a function of correcting distortion of the gaze position AT of the user U using the gaze position AT of the user U detected after a predetermined time (for example, 500 ms) has elapsed since the target point TG was displayed on the monitor screen 22.

[0021] The correction unit 11 has a function of determining whether the gaze position of the user U remains within a threshold area of ​​the monitor screen 22 for a predetermined period of time (e.g., 500 ms), and correcting the distortion of the gaze position AT of the user U using the gaze position AT of the user U that falls within the threshold area.

[0022] The correction unit 11 has a function of generating a conversion equation that converts the distortion of the gaze position AT before correction so that the distance between the multiple target points TG and each gaze position AT' after correction is minimized, and correcting the distortion of the gaze position AT of the user U using the conversion equation.

[0023] The correction unit 11 has a function of judging whether or not the correction of distortion of the gaze positions of multiple users U relative to multiple target points TG has all been successful based on a predetermined criterion, and if all have not been successful, automatically repeating the re-correction of distortion of the gaze positions AT of multiple users U.

[0024] The correction unit 11 has a function of repeating the re-correction of the distortion of the gaze positions AT of the multiple users U a predetermined number of times (for example, up to nine times).

[0025] The correction unit 11 has a function of, if a gaze position for which the correction has failed as a result of determining whether the correction has been successful or not based on a predetermined criterion, calculating an estimate of the gaze position before correction for the failed gaze position using each of the gaze positions before correction for the multiple gaze positions for which the correction has been successful, and setting the estimated value as a new gaze position before correction for the failed gaze position.

[0026] For example, if, after repeating the re-correction a predetermined number of times, only one of the corrections of the distortion of the gaze positions AT of multiple users U relative to multiple target points TG fails, the correction unit 11 uses the pre-correction gaze positions of the other multiple gaze positions that were successfully corrected to calculate an estimate of the pre-correction gaze position for that one correction, and sets this as a new pre-correction gaze position.

[0027] The determination unit 12 has a function of determining whether or not the tablet terminal 2 has normally detected the gaze position AT of the user U before the correction unit 11 corrects the distortion of the gaze position AT of the user U, or after the correction unit 11 fails to correct the distortion of the gaze position AT of the user U. For example, the determination unit 12 has a function of determining whether or not mirroring has occurred with respect to the gaze position AT of the user U detected by the tablet terminal 2.

[0028] The determination unit 12 has a function of causing the correction unit 11 to perform dummy correction when mirroring has occurred for the gaze position AT of the user U detected by the tablet terminal 2. The correction unit 11 has a function of performing dummy correction that only causes the detection unit 24 of the tablet terminal 2 to detect the gaze position of the user U when mirroring has occurred.

[0029] The notification unit 13 has a function of notifying the user U of instruction information related to the inclination of the face of the user U or instruction information related to the blinking of the user U by voice or text.

[0030] The target points are, for example, a plurality of target points displayed in a grid pattern at the center and periphery of the monitor screen 22. The number of target points is preferably five or more. For example, a total of five at the four peripheral corners including the center of the monitor screen 22, a total of five at the midpoints of each of the four surrounding sides including the center, a total of nine at the four peripheral corners including the center and the midpoints of each of the four surrounding sides, and a total of ten including the center in addition to the nine. The correction unit 11 corrects, for example, distortions of five or more gaze positions related to five or more target points. By setting the number of target points or the number of gaze positions to be corrected to five or more, the detection accuracy of the user's gaze position can be further improved.

[0031] The above-mentioned correction unit 11, determination unit 12, and notification unit 13 may be included in one functional unit (for example, one processing unit).

[0032] [Example of implementation of correction device 1] The correction device 1 may be implemented outside the tablet terminal 2. For example, it can be implemented in a server device on the Internet or in the cloud that is communicatively connected to the tablet terminal 2. In addition, the correction device 1 can be provided as a software program (including an application-type software program) that causes the tablet terminal 2 to function as the correction device 1.

[0033] [Overall operation] The following describes the operation performed when testing the eye movement function, etc., of a subject U, who is a user U. The subject U places his / her chin on a chin rest installed in front of the tablet terminal 2 and faces his / her face toward the camera 21 and monitor screen 22 of the tablet terminal 2.

[0034] FIG. 2 is a diagram showing an overall processing flow of the correction device 1 and the tablet terminal 2.

[0035] Step S101; The subject U or the examiner activates the camera 21 of the tablet terminal 2 .

[0036] Step S102; The detection unit 24 of the tablet terminal 2 detects the gaze position of the subject U from the face image of the subject U captured by the camera 21. The gaze position is the position of the subject U's gaze point relative to the monitor screen 22.

[0037] Step S103; The determination unit 12 of the correction device 1 acquires the eye position and gaze position of the subject U from the detection unit 24 of the tablet terminal 2, and determines whether or not the detection unit 24 has normally detected the gaze position of the subject U. For example, the determination unit 12 determines whether or not mirror imaging has occurred. If the detection unit 24 has not normally detected the gaze position of the subject U, the process returns to step S101, and step S102 is automatically repeated.

[0038] Step S104; When the detection unit 24 has normally detected the gaze position of the subject U, the correction unit 11 of the correction device 1 corrects the distortion of the gaze position of the subject U. The distortion of the gaze position is a deviation of the gaze position caused by the camera 21 photographing the subject U from the direction of the edge of the monitor screen 22.

[0039] Step S105; The correction unit 11 of the correction device 1 judges whether or not the correction of the distortion of the gaze position of the subject U has been successful based on a predetermined criterion. If the correction of the distortion of the gaze position of the subject U has been successful, the correction unit 11 outputs the corrected gaze position to an inspection device for eye movement function or the like or to an inspection unit in the tablet terminal 2, and ends the process. If the correction of the distortion of the gaze position of the subject U has failed, the process returns to step S103 and is automatically repeated.

[0040] In this way, the correction device 1 corrects the distortion of the gaze position of the subject U detected by the tablet terminal 2, thereby improving the detection accuracy of the gaze position of the subject U. The gaze position after the correction is used for the purpose of testing the eye movement function of the subject U, etc.

[0041] Such a correction device 1 can be applied to a diagnostic medical device that indicates the possibility of schizophrenia, such as a mental disorder, based on the results of various tests such as eye movement function and cognitive function. The correction device 1 can also be applied to fields other than the medical field.

[0042] Note that steps S104 and S105 may be executed before step S103. In this case, if the correction of the distortion of the gaze position is successful in one attempt in step S104, it becomes unnecessary to execute step S103, and therefore the correction processing time can be shortened.

[0043] [Gaze position distortion correction process] Next, the process of correcting the distortion of the gaze position, which is performed in step S104 in FIG. 2, will be described.

[0044] FIG. 3 is a diagram showing a flow of processing for correcting distortion of the gaze position.

[0045] Steps S201, S202; The correction unit 11 of the correction device 1 executes calibration. If the first calibration fails, the correction unit 11 executes a second calibration.

[0046] Specifically, the correction unit 11 displays multiple target points (e.g., 10 points) on the monitor screen 22 of the tablet terminal 2, and derives a conversion formula for correcting the distortion of each gaze position of the subject U detected by the detection unit 24 of the tablet terminal 2 to the correct gaze position for multiple target points (e.g., 9 points) excluding target points at the same position, and evaluates the conversion formula.

[0047] Steps S203, S204; If the calibration is successful in step S201 or step S202, the correction unit 11 of the correction device 1 executes validation. If the first validation fails, the correction unit 11 executes a second validation.

[0048] Specifically, the correction unit 11 displays multiple target points on the monitor screen 22 of the tablet terminal 2, similar to calibration, corrects the distortion of each gaze position of the subject U using the conversion formula derived during calibration, and determines that the correction is successful if the distances between each target point are all below a threshold value.

[0049] Step S205; If the validation is successful in step S203 or step S204, that is, if the correction of the distortion of each gaze position for all nine target points is successful, the correction unit 11 of the correction device 1 determines whether or not the elapsed time from the start of the first calibration to the present exceeds a threshold value (e.g., 70 seconds). If the elapsed time does not exceed the threshold value, the correction unit 11 normally ends the process.

[0050] Step S206; The correction unit 11 of the correction device 1 determines the number of retries for the entire inspection if calibration fails in step S202, if validation fails in step S204 (if correction is not successful at even one point), or if the elapsed time up to the present time exceeds a threshold in step S205.

[0051] Step S207; If the number of retries for the entire test is 1 to 8, the notification unit 13 of the correction device 1 outputs an error message indicating that the correction has failed to the monitor screen 22 of the tablet terminal 2, or outputs the error message by voice from the speaker 23. After that, the process returns to step S201. More precisely, the process returns to step S103 in FIG. 2 and resumes from the process of determining whether the detection unit 24 has normally detected the gaze position of the subject U.

[0052] Step S208; When the number of retries for the entire test is nine, the correction unit 11 of the correction device 1 determines whether or not the correction of eight out of nine points has been successful at least once. If the correction of eight points has not been successful at least once, the correction unit 11 abnormally ends the test.

[0053] Step S209; When the correction of the eight points has been successful at least once, the correction unit 11 of the correction device 1 estimates the gaze position before correction for the one point for which the correction has failed (one-point correction), generates a conversion formula for correcting the distortion of the gaze position based on the gaze position before correction, and corrects the distortion of the gaze position using the conversion formula. After that, the correction unit 11 normally ends the process.

[0054] [calibration] Next, the calibration performed in steps S201 and S202 in FIG. 3 will be described.

[0055] FIG. 4 is a diagram showing a processing flow of the calibration.

[0056] Step S301; The correction device 1 causes the tablet terminal 2 to display the ten target points one by one on the monitor screen 22 of the tablet terminal 2.

[0057] 5, the correction device 1 displays ten target points in a grid pattern, one at a time, at the center and on the periphery of the monitor screen 22. The first target point TG1 is displayed at the center, and after the first target point TG1 is deleted, the second target point TG2 is displayed at the upper left.

[0058] Similarly, the third to ninth target points TG3 to TG9 are displayed in the following order: top right → bottom right → bottom left → top center → right center → bottom center → left center. Finally, the tenth target point TG10 is displayed in the center.

[0059] That is, the correction device 1 first displays the central point, then displays the peripheral points, and finally displays the central point again.

[0060] The display order of the target points may be random. For example, the first target point may be displayed in the center, the second to ninth target points may be displayed in random order, and the tenth target point may be displayed in the center again. Alternatively, the first target point may be displayed in the upper left, and the tenth target point may be displayed in the upper right.

[0061] Step S302; At this time, the subject U gazes at each of the target points TG1 to TG10 in turn. The detection unit 24 of the tablet terminal 2 detects the gaze position of the subject U. Thereafter, the correction unit 11 of the correction device 1 acquires the coordinates of the gaze position of the subject U (coordinates of the gaze point) from the detection unit 24.

[0062] Here, a method for acquiring the coordinates of the gaze position will be described.

[0063] As shown in FIG. 6, the detection unit 24 of the tablet terminal 2 starts detecting the gaze position after a certain period (for example, 500 ms) has elapsed since the nth (n is a natural number) target point was displayed.

[0064] Then, the correction unit 11 of the correction device 1 acquires a group of xy coordinates of the gaze position of the subject U (a group of xy coordinates of the gaze point) detected within a certain gaze period (e.g., 500 ms) after the subject U starts gazing at the nth target point.

[0065] After that, the correction unit 11 performs gaze determination as to whether or not the xy coordinate group of the acquired gaze position falls within a preset threshold value. For example, as shown in Fig. 7, after 500 ms has elapsed since the display of the nth target point, for the xy coordinate group of the gaze position detected within a gaze period of 500 ms, a rectangle circumscribing the xy coordinate group of the gaze position is calculated, and it is determined whether or not the diagonal d of the rectangle is equal to or smaller than a threshold value.

[0066] After 500 ms has elapsed since the target point was displayed, gaze judgment is performed every 100 ms for the xy coordinate group of the gaze position within the gaze period of every 500 ms, and this is repeated until the gaze judgment is successful. If the diagonal d of the rectangle is below the threshold, the gaze judgment is successful.

[0067] Finally, the correction unit 11 of the correction device 1 calculates the median of the x and y coordinates of the gaze position during the gaze period in which the gaze determination was successful, and sets the median value as the x and y coordinates of the gaze position of the subject U.

[0068] As a result, the x and y coordinates (median values) of the gaze positions of 10 points are obtained for the 10 target points. However, since the positions of the first and tenth target points are the same, the x and y coordinates (median values) of the gaze positions of the nine target points from the second to the tenth are used.

[0069] Step S303; The gaze position acquired in step S302 is the center point of the xy coordinate group of gaze positions during the gaze period, but it is the gaze position detected by the tablet terminal 2, and there is a distortion in the gaze position due to the fact that the camera 21 photographed the subject U from the direction of the edge of the monitor screen 22.

[0070] Specifically, the orientation of the camera 21 (shooting direction) and the orientation of the subject U (gaze direction) are not on the same axis, and the lens size of the camera 21 is much smaller than the face size of the subject U, so distortion occurs in the gaze position of the subject U detected by the tablet terminal 2. In addition, the degree of distortion is greater for gaze positions farther from the camera 21 than for gaze positions closer to the camera 21.

[0071] Therefore, in the calibration and validation, the distortion of the gaze position of the subject U is corrected by polynomial fitting. In the calibration, a conversion equation is derived as the polynomial.

[0072] The correction unit 11 of the correction device 1 derives a conversion formula for converting the x and y coordinates (median) of the nine gaze positions of subject U acquired in step S302 so that the distance between the x and y coordinates of the nine target points displayed in step S301 and the x and y coordinates of the gaze position of subject U after conversion (correction) is minimized.

[0073] FIG. 8 is a diagram showing a flow of deriving the conversion formula.

[0074] Step S303(1); The correction unit 11 calculates the median (x, y) of the xy coordinate group of the gaze position during the gaze period in which the gaze determination was successful for each of the nine target points. The calculation of the median has already been explained, but will be described again. At this time, the correction unit 11 also calculates the center point (x_center, y_center) of the monitor screen 22 of the tablet terminal 2. Note that the coordinate value of the tenth target point may be used as the center point.

[0075] Step S303(2); The correction unit 11 converts each of the median values ​​(x, y) (raw coordinates) of the nine gaze positions into a coordinate system whose origin is the center of the monitor screen 22 of the tablet terminal 2. Specifically, the median point (x_center, y_center) is subtracted from each of the raw coordinates (x, y) of the nine points.

[0076] Step S303(3); The correction unit 11 defines, for example, the following conversion equations (1) and (2), with the x and y coordinates (raw coordinates after coordinate system transformation) of the gaze position of the subject U before transformation as x and y, and the x and y coordinates of the gaze position of the subject U after transformation as X and Y, and performs calculations using the least squares method.

[0077] X=a00×x+a01×y+a02×x×y+a03×x 2 +a04×y 2 +a05×(x×y) 2 +a06 …(1) Y=a10×x+a11×y+a12×x×y+a13×x 2 +a14×y 2 +a15×(x×y)2 +a16 …(2) The coefficients a00 to a06 and a10 to a16 in the conversion equations (1) and (2) are determined by the least squares method so that the conversion of the gaze position is optimal (i.e., so that the converted x and y coordinates are closest to the x and y coordinates of the target point). x and y are each fitted with a two-dimensional polynomial.

[0078] Step S304; The correction unit 11 evaluates the conversion formula derived in step S303.

[0079] Specifically, the correction unit 11 substitutes the coordinates (x, y) of the nine gaze positions before conversion acquired in step S302 into the conversion formula derived in step S303 to calculate the coordinates (X, Y) of the nine gaze positions after conversion, and calculates the distances from the coordinates of the nine target points (errors from the coordinates of the gaze positions).

[0080] The correction unit 11 judges the distance as Good if it is less than half the threshold for fair judgment, judges the distance as Fair if it is less than the threshold for fair judgment, and judges the distance as Poor if it exceeds the threshold for fair judgment.

[0081] Then, if the error judgment results for all nine points are within Fair (Good or Fair), the correction unit 11 judges the conversion formula to be OK and the calibration to be successful, but if even one point is Poor, it judges the conversion formula to be NG and determines that re-detection and re-correction of all nine points is necessary.

[0082] [Validation] Next, the validation performed in steps S203 and S204 in FIG. 3 will be described.

[0083] FIG. 9 is a diagram showing a process flow of validation.

[0084] Step S401; This is the same as step S301 in FIG.

[0085] Step S402; This is the same as step S302 in FIG.

[0086] Step S403; The correction unit 11 of the correction device 1 converts the coordinates of the gaze position of the subject U acquired in step S402 using the conversion equation determined to be a successful calibration in step S304 of FIG.

[0087] Step S404; As in step S304, the correction unit 11 of the correction device 1 calculates the distance (error from the coordinate of the gaze position) from the coordinates of the nine target points to the coordinate of the gaze position of subject U after transformation, and determines whether the validation is successful or not based on the length of that distance (accuracy of the error).

[0088] FIG. 10 is a diagram illustrating an example of validation state transition.

[0089] In the validation state V0, the distance between the coordinates of the target point and the coordinates of the transformed gaze position are calculated for each gaze position, and if all nine target points are within Fair (Good or Fair), the validation is judged to be successful and the state transitions to OK. If even one target point remains as Poor, the state transitions to validation state V1 as a retry target.

[0090] In validation state V1, only the target point to be retried is redisplayed, the coordinates of the transformed gaze position are recalculated, and the distance between the coordinates of the target point and the coordinates of the transformed gaze position are calculated. If the distance is within Fair (Good or Fair), validation is determined to be successful and the state transitions to OK.

[0091] On the other hand, if even one target point that has not been successfully validated remains after the retry, the validation is judged to have failed and the state transitions to NG. In this case, it is judged that redetection and recorrection of all nine points are necessary.

[0092] When retrying, each target point may be displayed in a random order and position. When retrying multiple times for only one remaining point, the target point is turned off and then turned on before the next display.

[0093] [One-point correction method] Next, the one-point correction performed in step S209 in FIG. 3 will be described.

[0094] 3, if even one gaze position that has failed validation remains, the process proceeds to step S206. If the number of retries for the entire test is small, an error message is notified in step S207, and the process returns to step S201 (more precisely, returns to step S103 shown in FIG. 2), and the correction of distortions of all nine gaze positions is re-executed.

[0095] On the other hand, if the number of retries for the entire test increases, it may be difficult to correct the gaze position distortion for all nine points even if the correction is repeated. In this case, the correction of the gaze position distortion may be immediately terminated, but if the correction is successful for many of the nine points, it is preferable to use the data at that time.

[0096] Therefore, when the number of retries for the entire test is nine, even if calibration and validation have failed, if there is an instance where only one point does not meet the standard, the gaze position of that one point that does not meet the standard is estimated from the other points, a conversion formula is generated, and the distortion of the gaze position is corrected using that conversion formula.

[0097] For example, as shown in FIG. 11, the corrected gaze positions for nine target points TG2 to TG10 are AT2' to AT10', and only AT3' is determined to be Poor.

[0098] In this case, the corrector 11 of the correction device 1 deletes the pre-correction gaze position AT3 determined to be Poor, as shown in the upper part of FIG.

[0099] Then, the correction unit 11 of the correction device 1 draws a straight line L1 passing through the two pre-conversion (pre-correction) gaze positions AT2, AT6 in the same row, as shown in the lower part of Figure 12, and draws a straight line L2 passing through the two pre-conversion gaze positions AT7, AT4 in the same column, and sets the intersection of the two straight lines L1, L2 as the new pre-conversion gaze position AT3.

[0100] Thereafter, the corrector 11 of the correcting device 1 derives a conversion equation in the same manner as in step S303 of FIG. 4, and corrects the distortions of the pre-correction gaze positions AT2 to AT10 using the conversion equation in the same manner as in step S403 of FIG.

[0101] The correction unit 11 of the correction device 1 may calculate a new pre-conversion gaze position AT3 by using a line (L3) passing through the other two gaze positions AT5 and AT10, instead of the line L1 or line L2. Additionally, the correction unit 11 may select one of the intersection point between the line L1 and the line L2, the intersection point between the line L1 and the line L3, and the intersection point between the line L2 and the line L3.

[0102] In addition, if there are multiple instances in which only one point out of the nine retries does not meet the standard, the data from the most recent instance will be used.

[0103] So far, the case where one-point correction has failed has been described as an example, but the present invention is applicable not only to the case where one-point correction has failed, but also to the case where two-point or three-point correction has failed. In other words, when there is a gaze position where correction has failed, an estimate of the gaze position before correction for the failed gaze position is calculated using each gaze position before correction of the multiple gaze positions where correction has been successful, and the estimate is set as a new gaze position before correction for the failed gaze position.

[0104] [Calibration or validation notification] The notification unit 13 of the correction device 1 displays the next teaching information in text on the monitor screen 22 of the tablet terminal 2 based on the detection results of the facial image of the subject U detected by the tablet terminal 2 before, during, or after the start of calibration or validation, depending on the progress of the test, and outputs the teaching information as audio from the speaker 23 of the tablet terminal 2.

[0105] For example, the notification unit 13 outputs instruction information such as, "Now, we will begin adjusting the machine. I would like you to look at the dots that appear on the screen. Please look at the dots that appear one after another without moving your head. When you look at the dots, stop blinking for a moment and keep looking until the dots disappear. Now, we will begin. Please look at the dot in the middle."

[0106] For example, the notification unit 13 outputs instruction information such as "Let's try again. Look at the dot in the middle. After that, a dot will appear on the edge of the screen, so do not close your eyes and keep staring at the dot until it disappears. Now, let's start. Look at the dot in the middle."

[0107] For example, the notification unit 13 outputs instruction information such as, "Let's do this again. I want you to look at the dots that appear on the screen. Please look at the dots that appear one after another without moving your head. When you look at the dots, stop blinking for a moment, do not close your eyes, and keep staring until the dots disappear. Now, let's start. Look at the dot in the middle."

[0108] For example, the notification unit 13 outputs instruction information such as "It appears that you are not looking clearly at the center of the dot. While the dot is displayed, please keep your eyes wide open and stare at it until the dot disappears. Relax for a moment, place your hands on the desk again, and rest your chin on the chin rest as if you are looking at the screen, and assume a comfortable position. Please refer to the diagram on the right for the angle from the screen."

[0109] [Processing to determine whether gaze position is correctly detected] Next, the process of determining whether the gaze position is normally detected, which is performed in step S103 in FIG. 2, will be described.

[0110] There are cases where the gaze position of the subject U cannot be detected normally by the tablet terminal 2 due to some reason. In this case, steps S104 and S105 in Fig. 2 cannot be executed appropriately, and the subsequent eye movement function test and the like cannot proceed normally. Therefore, when some abnormality occurs, the correction device 1 temporarily suspends the test using a certain criterion and performs the test again.

[0111] In this embodiment, as an example of some abnormality, a case where mirror imaging occurs on the tablet terminal 2 will be described. If mirror imaging is detected before or after the start of steps S104 and S105 in Fig. 2, the test is interrupted as if no test had been performed that time. After a break, steps S104 and S105 are performed again for the same time.

[0112] If mirroring occurs consecutively a threshold number of times (e.g., three times), the inspection is terminated abnormally. Even if mirroring does not occur consecutively the threshold number of times, if mirroring occurs a cumulative threshold number of times throughout the entire inspection (e.g., a cumulative total of three times), the inspection is terminated abnormally.

[0113] Here, the mirroring will be described.

[0114] As shown in Fig. 13, the tablet terminal 2 would normally detect the subject U at a position on the front side of the tablet terminal 2. Specifically, it would detect the subject U at a position on the "-" side on the z axis, which is the depth direction of the monitor screen 22. However, there are cases where the subject U is detected as subject U' at a position on the "+" side on the z axis. This phenomenon is called mirror imaging.

[0115] When such a mirror image occurs, the position of the subject U is misidentified, making it difficult to properly handle the gaze position of the subject U. Therefore, the position of the head, etc. is changed and the camera 21 of the tablet terminal 2 is restarted to eliminate the mirror image.

[0116] FIG. 14 is a diagram showing a process flow for determining whether or not mirroring is performed.

[0117] Step S501; The subject U or the examiner activates the camera 21 of the tablet terminal 2. If the camera 21 has already been activated, step S501 is omitted.

[0118] Step S502; The determination unit 12 of the correction device 1 determines whether or not mirror imaging has occurred in the tablet terminal 2. For example, after the camera 21 is started, the determination unit 12 determines whether or not the value of the eye or head position on the z axis detected by the detection unit 24 of the tablet terminal 2 is greater than '0'.

[0119] Step S503; If the value of the eye or head position on the z-axis is greater than '0', the determination unit 12 of the correction device 1 determines that mirroring has occurred, and determines the number of consecutive occurrences of mirroring.

[0120] Step S504; When the number of consecutive occurrences of mirroring is one, the determination unit 12 of the correction device 1 determines the cumulative number of occurrences of mirroring throughout the entire inspection.

[0121] Step S505; When the number of consecutive occurrences of mirroring is two, and when the cumulative number of occurrences of mirroring throughout the entire examination is one, the determination unit 12 of the correction device 1 instructs the correction unit 11 to acquire the gaze position of the subject U and to correct the distortion of the gaze position of the subject U (dummy correction). At this time, the correction unit 11 corrects the distortion of the gaze position of the subject U (dummy correction). The dummy correction will be described later.

[0122] At this time, the notification unit 13 of the correction device 1 notifies the user of instruction information in text or voice, for example, "Look at the dots appearing on the screen. Look at the dots that appear one after another without moving your head. When you look at the dots, stop blinking for a moment and keep looking until the dots disappear. Now we will begin. Look at the dot in the middle."

[0123] Step S506; After step S505, the notification unit 13 of the correction device 1 instructs the subject U by text or voice to change the position of his / her head in order to eliminate the mirror image.

[0124] For example, the notification unit 13 notifies the user in text or voice with instruction information such as, "Your face may have moved. Relax for a moment, place your hands on the desk again, place your chin on the chin rest as though you are looking at the screen, and assume a comfortable position. Refer to the diagram on the right for the angle from the screen."

[0125] After that, the process returns to step S501. More precisely, the process returns to step S101 in Fig. 2. Then, the camera is restarted in step S501, and it is determined again in step S502 whether or not mirror imaging has occurred.

[0126] Step S507; If the number of consecutive occurrences of mirroring is three, or if the cumulative number of occurrences of mirroring throughout the entire examination is two, the judgment unit 12 of the correction device 1 corrects (performs dummy correction) the distortion of the gaze position of the subject U, as in step S505.

[0127] Step S508; After step S507, the determination unit 12 of the correction device 1 determines that it is difficult to eliminate the mirror image on the tablet terminal 2, and ends the inspection abnormally.

[0128] Note that the dummy correction performed in step S505 or step S507 is actually a measurement of the apparent gaze position that does not acquire the desired data, that is, a process that simply causes the detection unit 24 of the tablet terminal 2 to detect the gaze position of the subject U.

[0129] Since the mirror image determination is completed quickly, it gives the subject U the impression that the test of eye movement function, etc., ended suddenly, but by performing dummy correction in steps S505 and S507, it is possible to reduce the mental burden on the subject U from the test. However, in the dummy correction, it is also possible to actively correct the distortion of the gaze position of the subject U using the detected gaze position of the subject U.

[0130] Step S509; If the value of the eye or head position on the z-axis is equal to or less than '0' in step S502, the determination unit 12 of the correction device 1 determines that mirroring has not occurred, and ends the process normally.

[0131] The above-mentioned number of consecutive occurrences and the cumulative number of occurrences of mirroring are examples. For example, the number of consecutive occurrences may be 5, and the cumulative number of occurrences may be 7. In addition, the presence or absence of mirroring is determined within a time period of, for example, 0.5 seconds.

[0132] [effect] According to this embodiment, the correction unit 11 acquires each gaze position AT of the user U detected by the tablet terminal 2 when the user U, photographed by the camera 21, gazes at each of the multiple target points TG displayed sequentially on the monitor screen 22, and corrects the distortion of the gaze position AT of the user U based on the fact that the camera 21 photographed the user U from the direction of the edge of the monitor screen 22, thereby providing a technology that can improve the detection accuracy of the gaze position.

[0133] According to this embodiment, the correction unit 11 determines whether the correction is successful or unsuccessful based on a predetermined criterion, and if there is a gaze position for which the correction fails, an estimate of the pre-correction gaze position AT for the failed gaze position is calculated using each pre-correction gaze position AT of the multiple gaze positions for which the correction was successful, and the estimate is set as a new pre-correction gaze position AT for the failed gaze position, thereby providing a technology that can further improve the detection accuracy of the gaze position.

[0134] According to this embodiment, the correction unit 11 corrects the distortion of the gaze position AT of the user U using the gaze position AT of the user U detected 500 ms after the target point TG is displayed on the monitor screen 22, thereby providing a technology that can further improve the detection accuracy of the gaze position.

[0135] According to this embodiment, the correction unit 11 determines whether the gaze position of the user U falls within the threshold area of ​​the monitor screen 22 continuously for 500 ms, and corrects the distortion of the gaze position AT of the user U using the gaze position AT of the user U that falls within the threshold area, thereby providing a technology that can further improve the detection accuracy of the gaze position.

[0136] According to this embodiment, the correction unit 11 generates a conversion formula that converts the distortion of the pre-correction gaze position AT so that the distance between the multiple target points TG and each corrected gaze position AT' is minimized, and corrects the distortion of the gaze position AT of the user U using the conversion formula, thereby providing a technology that can further improve the detection accuracy of the gaze position.

[0137] According to this embodiment, the correction unit 11 corrects five or more distortions in the position of the user's gaze, and therefore it is possible to provide a technique that can further improve the detection accuracy of the position of the gaze.

[0138] According to this embodiment, the multiple target points are five or more target points displayed in a grid pattern in the center and on the periphery of the screen, and the central point is displayed followed by the peripheral points, thereby providing a technology that can further improve the accuracy of detecting the gaze position.

[0139] According to this embodiment, the judgment unit 12 judges whether the tablet terminal 2 has correctly detected the gaze position AT of the user U before the correction unit 11 corrects the distortion of the gaze position AT of the user U, or after the correction unit 11 fails to correct the distortion of the gaze position AT of the user U, thereby providing a technology that can further improve the detection accuracy of the gaze position.

[0140] According to this embodiment, when mirroring occurs in the gaze position of user U detected by tablet terminal 2, the correction unit 11 performs a dummy correction that only causes tablet terminal 2 to detect the gaze position of user U, thereby reducing the mental stress on subject U regarding the examination.

[0141] According to this embodiment, the notification unit 13 notifies the user U of instruction information regarding the inclination of the user U's face or instruction information regarding the user U's blinking, thereby providing a technology that can further improve the detection accuracy of the gaze position.

[0142] [others] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the present invention.

[0143] The correction device 1 of the present embodiment described above can be realized, for example, by using a general-purpose computer system including a CPU 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 15. The memory 902 and the storage 903 are storage devices. In the computer system, the CPU 901 executes a predetermined program loaded on the memory 902, thereby realizing each function of the correction device 1.

[0144] The correction device 1 may be implemented in one computer. The correction device 1 may be implemented in multiple computers. The correction device 1 may be a virtual machine implemented in a computer. The program for the correction device 1 may be stored in a computer-readable recording medium such as an HDD, SSD, USB memory, CD, or DVD. The computer-readable recording medium is, for example, a non-transitory recording medium. The program for the correction device 1 may also be distributed via a communication network. [Explanation of symbols]

[0145] 1 Correction device 11 Correction section 12 Judgment section 13 Notification Department 2. Tablet devices 21 Camera 22 Monitor screen 23 Speaker 24 Detection unit 25 Memory section 901...CPU 902…Memory 903…Storage 904...Communication equipment 905…Input device 906...Output device

Claims

1. a correction unit that acquires each gaze position of the user detected by a computer when the user photographed by a camera gazes at each of a plurality of target points displayed sequentially on a screen, and corrects distortion of the gaze position of the user based on the fact that the camera photographed the user from a specific direction; The correction unit is A correction device that determines whether the correction is successful or unsuccessful based on predetermined criteria, and if there is a gaze position for which the correction fails, calculates an estimate of the pre-correction gaze position for the failed gaze position using each of the gaze positions before correction for the multiple gaze positions for which the correction was successful, sets the estimate as a new pre-correction gaze position for the failed gaze position, and corrects the distortion of the user's gaze position using the new pre-correction gaze position.

2. The correction unit is The correction device according to claim 1 , further comprising: a conversion equation that converts a distortion of the gaze position before correction so as to minimize a distance between the plurality of target points and each gaze position after correction; and a correction device that corrects the distortion of the gaze position of the user using the conversion equation.

3. The correction unit is The correction device according to claim 1 , which determines whether the user's gaze position remains within a threshold area of ​​the screen for a predetermined period of time, and corrects distortion of the user's gaze position using the user's gaze position that falls within the threshold area.

4. The correction unit is The correction device according to claim 1 , which corrects distortion of the user's gaze position corresponding to five or more target points.

5. The plurality of target points are 2. The correction device according to claim 1, wherein five or more target points are displayed in a grid pattern at the center and on the periphery of the screen, with the center point being displayed first, followed by the peripheral points.

6. a determination unit that determines whether or not the computer has normally detected the user's gaze position before correcting the distortion of the user's gaze position or after the correction of the distortion of the user's gaze position has failed, The determination unit is The correction device according to claim 1 , wherein the computer determines whether or not mirror imaging has occurred for the detected gaze position of the user.

7. The correction unit is The correction device according to claim 6 , wherein, when mirror imaging occurs in the gaze position of the user detected by the computer, a dummy correction is performed in which the computer only detects the gaze position of the user.

8. a notification unit that notifies the user of instruction information regarding the inclination of the user's face or instruction information regarding the user's blinking; The correction device of claim 1 further comprising:

9. In a correction method performed by a correction device, acquiring each gaze position of the user detected by a computer when the user photographed by the camera gazes at each of a plurality of target points displayed sequentially on a screen, and correcting distortion of the gaze position of the user based on the fact that the camera photographed the user from a specific direction; determining whether the correction has been successful or unsuccessful based on a predetermined criterion, and if there is a gaze position for which the correction has failed, calculating an estimate of the gaze position before correction for the failed gaze position using each of the gaze positions before correction for the multiple gaze positions for which the correction has been successfully performed, setting the estimate as a new gaze position before correction for the failed gaze position, and correcting the distortion of the gaze position of the user using the new gaze position before correction; Correction method.

10. A correction program that causes a computer to function as the correction device according to any one of claims 1 to 8.

Images