Eye-tracking device and method

Abstract

A device including at least one camera (101), a display (103), a memory (105), and a processor (104), wherein the memory stores instructions that, when executed by the processor, cause the device to: use the device's display to display content to be viewed by a user; use the device's camera to detect the user's eyes; use the device's camera to detect the user's eye movements; and determine whether the user has viewed the content based on the detected user's eye movements.

Description

[Technical Field]

[0001] EYE TRACKING DEVICE AND METHODS FIELD OF THE INVENTION The present invention relates to eye tracking devices and methods, which are used, for example, for supervised training. [Background technology]

[0002] Users of various drug delivery devices, such as insulin pens, need to be trained in the use of the device. Such training is particularly necessary and / or beneficial when a user is first issued a particular device (e.g., a newly diagnosed patient, or a user who has been using a drug delivery device for some time but is being encouraged to switch to a different, unfamiliar device, a different drug, and / or a different dosing regimen).

[0003] For various reasons, it may be useful or necessary to supervise training and / or ensure that users complete all required sections of training, for example, a regulatory body may require that users, manufacturers, or both be issued a certificate of user completion of standardized training.

[0004] Currently, drug delivery device training is typically provided by healthcare professionals (e.g., nurses). Such training is time-consuming and inefficient for healthcare professionals. Furthermore, it is difficult for healthcare professionals to ensure that users have understood all the required training content. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, there is a need to provide automated and standardized training. [Means for solving the problem]

[0006] In a first aspect, a device is provided, the device including at least one camera, a display, a memory, and a processor, the memory storing instructions that, when executed by the processor, cause the device to: use the display of the device to display content to be viewed by a user; use the camera of the device to detect the user's eyes; use the camera of the device to detect the user's eye movements; and determine whether the user has viewed the content based on the detected user's eye movements.

[0007] In a second aspect, a device is provided, the device including at least one camera, a motion sensor, a display, a memory, and a processor, the memory storing instructions that, when executed by the processor, cause the device to: display content to be viewed by a user using the device's display; detect the user's eyes using the device's camera, the device's motion sensor, or both the camera and the motion sensor; detect the user's eye movements using the device's camera, the device's motion sensor, or both the camera and the motion sensor; and determine whether the user has viewed the content based on the detected user's eye movements.

[0008] In some embodiments, the first and second aspects are combined with one or more of the following configurations: the at least one camera includes at least one front-facing camera; Detecting the user's eye movements: the direction of the user's gaze, the speed of the user's eye movements , including detecting one or more of the direction of the user's eye movement, the time spent by the user concentrating on the content, the timing of the user's blinks, the rate at which the user blinks, the duration of pupil fixations, the number of pupil fixations, the gaze path, the pupil diameter and stress load, and / or dwell time; Determining whether the user has viewed the content includes monitoring the duration of the user's eye blinks; Determining whether the user has viewed the content further includes comparing the value to a pre-stored average value of eye blink duration; If the duration of the user's eye blink is greater than 0.5 seconds, the user is determined not to have viewed the content; Determining whether the user has viewed the content further includes comparing the value with a pre-stored average value of eye blink duration, and if the duration of the user's eye blink is greater than 0.5 seconds, it is determined that the user has not viewed the content; Determining whether the user has viewed the content includes monitoring the user's eye blink rate; Determining whether the user has viewed the content further includes comparing the value to a pre-stored average value of blink rates; Determining whether the user has viewed the content includes monitoring an eye blink rate of the user, and determining whether the user has viewed the content further includes comparing the value to a pre-stored average value of the eye blink rate; Determining whether the user has viewed the content includes monitoring the pupil diameter of the user's eye; Determining whether the user has viewed the content further includes comparing the value to a pre-stored value of pupil diameter; Determining whether the user has viewed the content includes monitoring a pupil diameter of the user's eye, and determining whether the user has viewed the content further includes comparing the value to a pre-stored value of pupil diameter; Determining whether the user has viewed the content includes monitoring a number of eye fixations; The content includes text, and determining whether the user has viewed the content further includes comparing the number of eye fixations to the number of words in the text; If the number of eye fixations corresponds to 80%-100% of the number of words in the text, the user is determined to have viewed the content; Determining whether the user has viewed the content includes monitoring a number of eye fixations, and when the content includes text, determining whether the user has viewed the content further includes comparing the number of eye fixations to a number of words in the text; Determining whether the user has viewed the content includes monitoring a number of eye fixations, and when the content includes text, determining whether the user has viewed the content further includes comparing the number of eye fixations with a number of words in the text, and if the number of eye fixations corresponds to 80% to 100% of the number of words in the text, it is determined that the user has viewed the content; Determining whether the user has viewed the content includes monitoring the duration of eye fixations; If the duration of the user's eye fixations is between 200 milliseconds and 250 milliseconds for at least 80% of the fixations, the user is determined to have viewed the content; The device further includes a motion sensor, the device being adapted to detect eye movements of the user using the camera and the motion sensor of the device.

[0009] In a third aspect, a method for determining whether a user has viewed content is provided, the method including: displaying content to be viewed by the user; detecting eye movements of the user; and determining whether the user has viewed the content based on the detected eye movements of the user.

[0010] The method of the third aspect uses one or more of the following specific configurations: Determining whether the user has viewed the content includes monitoring the user's eye blink rate; Determining whether the user has viewed the content includes monitoring the pupil diameter of the user's eye; Determining whether the user has viewed the content includes monitoring a number of eye fixations; Determining whether the user has viewed the content includes monitoring the duration of eye fixations.

[0011] The following description refers to the following figures: [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an exemplary system of a mobile device, augmented reality, virtual reality, or mixed reality glasses, and a training drug delivery device. [Figure 2] FIG. 1 shows a schematic example of supervised reading. [Figure 3] FIG. 1 is a schematic diagram of the electronic components of a user device. [Figure 4] 1 is a flowchart illustrating a method for supervising the training of a user. DETAILED DESCRIPTION OF THE INVENTION

[0013] 1 shows a training system including augmented reality (AR), virtual reality (VR), or mixed reality (MR) glasses 12, a user device 10, and a training drug delivery device 14. The AR, VR, or MR glasses 12 will hereinafter be referred to as glasses 12. When the system includes glasses 12, the user device 10 is optional. When the system includes user device 10, the glasses 12 is optional. The training drug delivery device 14 is optional.

[0014] The glasses 12 may be a head mount for a mobile device such as the user device 10. The system may include both the glasses 12 and the user device 10. The glasses 12 may be a device independent of the mobile device. If the glasses 12 are independent of the mobile device, the user device 10 is optional.

[0015] The user device 10 may be, for example, a mobile device such as a mobile phone or a tablet, or a desktop PC.

[0016] For the sake of brevity, the following will be described only in relation to the user device 10. It should be understood that in the following, the user device 10 may be replaced with the glasses 12 or with a system including the user device 10 and the glasses 12.

[0017] The user device 10 has a camera 101 associated with the user device 10. The camera 101 may be a front-facing camera of a mobile device, i.e., the camera is provided as a member on the same side of the user device 10 as the display 103. The camera 101 may be aimed at the user when the user device 10 is in use. The camera 101 may be a webcam or any other suitable external camera associated with the mobile device or a desktop PC.

[0018] In one embodiment, the user device 10 may include a light source associated with the user device 10. The device 100 includes a light source 102. The light source 102 can be provided as a component of the user device 10. The light source 102 can be provided as a separate device, in which case the light source 102 is preferably adapted to be attached to the user device 10 and / or the glasses 12. The light source 102 can be provided as a component of the glasses 12. The light source 102 can be an infrared (IR) light source. The light source 102 can be a laser light source.

[0019] The user device 10 has a display 103. In one embodiment, the IR light is additionally or exclusively provided by the display 103 of the user device 10. For example, if the display 103 is an LED or OLED display, it is possible to use the display 103 as a source of the IR light. The IR light may originate partially or exclusively from the display 103.

[0020] The user device 10 may include a motion sensor 107. The motion sensor may be, for example, a radar sensor 107.

[0021] The user device 10 further includes a processor 104, a memory 105, and a power supply 106. These components are shown in Figure 3 and further described below.

[0022] The user device 10 has a memory 105 that stores one or more suitable software applications for carrying out the methods described below.

[0023] Using the camera 101, the motion sensor 107, or both the camera 101 and the motion sensor 107, the user device 10 may monitor the user's eyes. The user device 10 may monitor the user's eye position. The user device 10 may monitor the user's eye movement. The user device 10 may use the camera 101 to monitor the user's eye position and the user's eye movement. The user device 10 may use the camera 10 to monitor the user's eye position and the motion sensor 107 to monitor the user's eye movement (e.g., the movement of the user's eyes and / or the user's eyelids, as described below).

[0024] In one embodiment, the user device 10 includes at least two front-facing cameras 101. The user device 10 may use the at least two cameras 101 to monitor the position and / or movement of the user's eyes. Alternatively, or in addition, the user device may use a motion sensor 107 to monitor the user's eye movements (e.g., the movement of the user's eyes and / or the user's eyelids). Using two cameras can enable providing a stereoscopic image, which can improve the accuracy and efficiency of measuring the user's eye movements. Using the motion sensor 107 can further improve the accuracy in detecting the user's eye movements (e.g., the movement of the user's eyes and / or the user's eyelids).

[0025] Hereinafter, user eye movement means one or more of: user's direction of gaze, user's eye movement speed, user's eye movement direction, time spent by the user concentrating on content (reading or looking), user's blink timing, user's blink speed, pupil fixation duration, number of pupil fixations, gaze path, pupil diameter and stress load; and dwell time (per content).

[0026] Hereinafter, monitoring the user's eye movements means monitoring one or more of the above listed user's eye movements.

[0027] The acquired data regarding the user's eye movements is used to determine whether the user spent enough time on particular training content. The acquired data regarding the user's eye movements is used to determine whether the user paid sufficient attention to the training content. Using the acquired measurements of the user's eye movements, the user device 10 can determine whether the user read or viewed all of the training content of a training. Using the acquired measurements of the user's eye movements, the user device 10 can determine whether the user understood all of the training content. Using the detected measurements of the user's eye movements, the user device 10 can determine the likelihood that the user understood the training content.

[0028] Examples of monitoring a user's eye movements using the camera 101 and / or motion sensor 107 and the user device 10 are shown in FIGS. 2a-2d. FIG. 2a shows training content 20 displayed on the display 103 of the user device 10. The training content 20 consists of blocks of text 201-206. To successfully complete the training, the user is required to read and understand all blocks of text 201-206. For example, the training content 20 may relate to the use of an insulin pen and may also include drug-related information, device-related information, contraindication information, etc.

[0029] As the user begins to read block 201, camera 101 and / or motion sensor 107 track the user's eye movements. The user's eyes gradually move along the lines of text in block 201, as shown by arrows 201a-201z in Figure 2b. The user's device 10 tracks the user's eye movements along these lines 201a-201z.

[0030] To determine whether the user has read and understood block 201, user device 10 may store information regarding the average time required to read and understand the text in block 201. For example, if the user spends the average time required to read and understand the text in block 201, or more time than the average time required to read and understand the text in block 201, user device 10 may indicate that the user has successfully completed block 201.

[0031] The user device 10 may use alternative or additional information, such as the user's eye movements (described above), to determine whether the user has read and understood the text in block 201. If the user spends less than the average time required to read and understand the text in block 201, the user device 10 may use further information, such as the user's eye movements, to determine whether the user has read and understood all of the training content in block 201.

[0032] The user device 10 may use alternative or additional information, such as the user's eye movements (described above), to determine whether the user has read and understood the text in the block 201, regardless of how long it takes the user to read the block 201.

[0033] For example, if the user requires the average time required to read and understand the text in block 201 and the pupil diameter indicates that the user was paying attention to the text, the user device 10 may indicate that block 201 was successfully completed.

[0034] As another example, if a user requires more than the average time required to read and understand the text in block 201, but pupil diameter indicates that the user was not paying attention to the text, user device 10 may indicate that block 201 was not successfully completed.

[0035] As another example, if a user requires less than the average time required to read and understand the text in block 201, but the pupil diameter indicates that the user was not paying attention to the text, the user device 10 may indicate that block 201 was not successfully completed.

[0036] In the example of Figure 2, after reading block 201, the user's eyes continue to block 203, as illustrated in Figure 2c. The user reads block 203. The success or failure of completing block 203 is evaluated and indicated in the same manner as block 201, as described above.

[0037] 2c, after reading block 203, the user continues to block 202 and reads the text within block 202. After reading block 202, the user continues to block 206 and reads block 206. The success or failure of completing blocks 202 and 206 is evaluated and indicated in the same manner as block 201, as described above.

[0038] In the example of FIG. 2c, the user does not read blocks 204 and 205, which are evaluated and presented in the same manner as block 201, as explained above.

[0039] 2d shows one possible indication of the results of an assessment of a user's reading and comprehension of information. Based on measurements of the user's eye movements (described above), the user device 10 determines that the user read and understood blocks 201, 202, and 206; that the user read but did not understand block 203; and that the user did not read 204 and 205. The user device 10 may then display information on the display 103 of the user device 10 indicating the status of each block 201-206.

[0040] The status of each block 201-206 is indicated by a different color (e.g., green for blocks 201, 203, 206 that the user has read and understood; red for blocks 204, 205 that the user has not read; and red or orange for block 203 that the user has read but not understood). Alternatively, or in addition, the status of each block 201-206 is indicated by a different texture, a different background image, a check / cross displayed over the lock 201-206, or any other suitable method.

[0041] Preferably, a training session is marked as completed when the user has read and understood all blocks 201-206. If the user does not read and understand all blocks 201-206, the user is allowed and / or prompted to return to the training session at a later time.

[0042] Alternatively, or in addition, the user device 10 may use the user's eye movements to monitor with content other than textual content. For example, the user may be required to view a training video in which the correct use of a drug delivery device is demonstrated and / or an avatar (e.g., simulating a medical professional) explains drug-related or drug delivery device-related information to the user. The user's attention must be focused on the drug delivery device and / or avatar shown in the video, as appropriate. The user's eye movements are monitored in a manner similar to that described above to evaluate whether the user's attention is focused on the correct object (e.g., the drug delivery device shown, the avatar shown, etc.). For example, if the user is determined to be looking elsewhere than at the drug delivery device or avatar for the majority of the duration of the training video, the user device 10 may determine that the user was not paying attention and that viewing of the video needs to be repeated.

[0043] A method for determining a user's comprehension of training content is illustrated in FIG.

[0044] The user device 10 comprises a processor 104 and a memory 105. The memory 105 stores instructions that, when executed by the processor 104, implement the following methods.

[0045] In step 401, the user device 10 displays training content using the display 103. In step 402, the user device 10 detects the user's eyes using the camera 101 and / or the motion sensor 107. In step 403, the device monitors the user's eye movements using the camera 101 and / or the motion sensor 107, for example, as described above in connection with FIGS. 2a-2d. In step 404, the user's comprehension of the training content displayed on the display 103 in step 401 is determined, for example, as described above in connection with FIGS. 2a-2d.

[0046] Monitoring the user's eye movements and / or determining whether the user has read and understood the content may include one or more of the following in any combination: While the following description is provided with a focus on text content, it should be understood that the same or similar considerations apply to image / video content.

[0047] The duration of a user's eye blink is measured. For example, the typical duration of a blink in an attentive reader (i.e., the time elapsed between the moment the eyelids are completely closed and the moment they begin to open again) is 0-0.2 seconds. The duration of a blink in a non-focused user is typically 0.3-0.5 seconds. The duration of a blink in a tired person is >0.5 seconds. By detecting the duration of the blink and comparing the value with a typical value such as those described above (e.g., a pre-stored average or a pre-stored standard for a particular user), the user device 10 can determine whether the user was paying attention to the text. In one embodiment, if the duration of the blink exceeds 0.5 seconds, the user device 10 interrupts the training.

[0048] The blink rate (i.e., the number of blinks per time unit) is measured. For example, a higher and / or increasing blink rate is an indication of attentiveness (the user is paying attention to the text). A lower and / or decreasing blink rate is an indication of the user not paying attention and / or the user lacking comprehension of the text. In one embodiment, the user device 10 may detect the user's blink rate and compare the measured value to a typical value, such as a pre-stored average, a pre-stored standard for a particular user, etc. In one embodiment, the user device 10 may interrupt training if the blink rate is at or below a threshold.

[0049] The pupil diameter of the user's eye is measured. Typically, an increased pupil diameter can serve as an indication that the user is paying attention to the text. For example, if the pupil diameter increases while viewing the text, the user device 10 may determine that the user is paying attention to the text.

[0050] The number and duration of eye fixations are measured. For example, a user who is paying attention to the text will fixate on at least 80% of the words. Thus, if the number of fixations corresponds to 80%-100% of the number of words, the user is determined to have paid attention and / or understood the text. In one embodiment, if the number of fixations is at least 80% of the number of words, the user is determined to have paid attention and / or understood the text. For example, the duration of the fixations is 200-250 milliseconds. If the duration of each fixation is consistently shorter than, for example, 200 milliseconds, the user is determined to have not paid attention. and / or not comprehending the text.

[0051] In one embodiment, blink rate, blink duration, and pupil diameter are all measured and used together to determine a user's level of attention, which, along with the user's direction of gaze, the user's eye movement speed, the user's eye movement direction, and the time spent by the user concentrating on reading the content, are used to determine whether the user paid attention to and / or understood the text.

[0052] In one embodiment, the user device 10 may provide the user with a question related to the text after pausing the exercise.

[0053] In one embodiment, the user's device 10 may cooperate with a training drug delivery device 14. The training drug delivery device 14 is a device that resembles the actual drug delivery device that the user will use (e.g., the device that has been prescribed for the user).

Claims

1. A device including at least two cameras (101), a motion sensor (107), a display (103), a memory (105), and a processor (104), wherein the memory stores instructions that, when executed by the processor: using a display of the device to display content to be viewed by the user, the content including training content related to the drug delivery device; detecting the user's eyes using at least two cameras of the device; detecting the user's eye movements using both at least two cameras and a motion sensor of the device; Determining whether a user has viewed content based on detected user eye movements where the detection of the user's eye movements is monitoring the position of a user's eyes with at least two cameras; monitoring the user's eye movements with a motion sensor; generating a stereoscopic image using at least two cameras of the device; and processing the stereoscopic image to detect eye movements of the user.

2. The device of claim 1 , wherein the at least two cameras include at least two front-facing cameras.

3. 3. The device of claim 1 or 2, wherein detecting the user's eye movements comprises detecting one or more of: a direction of the user's gaze, a speed of the user's eye movements, a direction of the user's eye movements, a time spent by the user concentrating on content, a timing of the user's blinks, a speed of the user's blinks, a duration of pupil fixations, a number of pupil fixations, a gaze path, a pupil diameter and stress load, and / or a dwell time.

4. The device of any one of claims 1 to 3, wherein determining whether the user has viewed the content comprises monitoring the duration of the user's eye blinks.

5. The device of any one of claims 1 to 4, wherein determining whether the user has viewed the content comprises monitoring the blink rate of the user's eyes.

6. The device of any one of claims 1 to 5, wherein determining whether the user has viewed the content comprises monitoring the pupil diameter of the user's eye.

7. The device of any one of claims 1 to 6, wherein determining whether a user has viewed content comprises monitoring a number of eye fixations.

8. The device of any one of claims 1 to 7, wherein determining whether the user has viewed the content comprises monitoring the duration of eye fixations.

9. 1. A method for determining whether a user has viewed content, comprising: displaying content to be viewed by the user, the content including training content related to the drug delivery device; detecting the user's eyes using at least two cameras of the device; detecting the user's eye movements using both at least two cameras and a motion sensor of the device; Determining whether a user has viewed content based on detected user eye movements wherein detecting the user's eye movements includes: monitoring the position of a user's eyes with at least two cameras; monitoring the user's eye movements with a motion sensor; generating a stereoscopic image using at least two cameras of the device; and processing the stereoscopic images to detect eye movements of the user.

10. The method of claim 9 , wherein determining whether the user has viewed the content includes monitoring the user's eye blink rate.

11. The method of claim 9 or 10, wherein determining whether the user has viewed the content comprises monitoring the pupil diameter of the user's eye.

12. The method of any one of claims 9 to 11, wherein determining whether a user has viewed content comprises monitoring a number of eye fixations.

13. The method of any one of claims 9 to 12, wherein determining whether the user has viewed the content comprises monitoring the duration of eye fixations.

Images