Head-mounted device for tracking screen time
A head-mounted device with multi-mode operation and sensors tracks screen time across devices, addressing measurement accuracy and power consumption issues, and notifies users to manage their screen time effectively.
Patent Information
- Application Number
- KR1020247036372
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-05-16
- Filing Date
- 2023-05-05
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2043-05-05
AI Technical Summary
Existing technologies struggle to accurately measure and aggregate screen time across multiple devices, and they often consume excessive power, limiting their usability and failing to notify users of their screen time effectively.
A head-mounted device equipped with various sensors and cameras, including eye-tracking and front-facing cameras, orientation sensors, and ultra-wideband sensors, operates in multiple power modes to track screen time across devices, conserving power when needed, and generates notifications based on predefined criteria.
Accurately measures and aggregates screen time across multiple devices while optimizing power consumption and providing timely notifications to users, helping them manage their screen time effectively.
Smart Images

Figure R1020247036372_ABST
Abstract
Description
Technology Field
[0001] Cross-reference regarding related applications
[0002] This application is a continuation of U.S. Application No. 17 / 663,444 filed on May 16, 2022, and claims the benefit thereof. The disclosure is incorporated herein by reference in its entirety.
[0003] The present disclosure relates to a head-mounted device, more specifically, a head-mounted device configured to measure screen time across a plurality of devices. Background Technology
[0004] Screen time refers to the amount of time a person spends using a screen-equipped device. Prolonged screen time can have negative effects on sleep, physical health, brain development, and / or behavior.
[0005] In some embodiments, the technology described herein relates to a head-mounted device, wherein the head-mounted device comprises: an eye-tracking camera directed toward a user's eye; and a processor communicating with the eye-tracking camera, wherein the processor is configured by instructions to: capture an eye image of the user's eye using the eye-tracking camera; analyze the eye image to identify a screen reflection; start a screen timer after a screen reflection is identified; periodically capture a subsequent eye image to track the screen reflection over time; stop the screen timer when the screen reflection is no longer tracked; and record a screen time of a first instance based on the screen timer.
[0006] In some embodiments, the technology described herein relates to a head-mounted device, wherein the step of analyzing an eye image to identify a screen reflection is performed by a neural network.
[0007] In some embodiments, the technology described herein relates to a head-mounted device, wherein the processor is further configured to: classify the screen time of a first instance of a first device based on a machine learning model configured by device information captured by the head-mounted device; and record the screen time of a first instance of a first device in a database.
[0008] In some embodiments, the technology described herein relates to a head-mounted device, wherein the device information is any combination of a device identifier, the content of a screen reflection, the characteristics of a screen reflection, and the location of a user.
[0009] In some embodiments, the technology described herein relates to a head-mounted device, wherein the processor is further configured by an instruction to detect that the mode of the head-mounted device for screen time tracking is a low-power mode.
[0010] In some embodiments, the technology described herein relates to a head-mounted device, further comprising a front camera, and a processor further configured to: detect that the mode of the head-mounted device for screen time tracking is a high-power mode; capture a field of view image of the user's field of view using the front camera; analyze the field of view image to identify a screen; start a screen timer after the screen is identified in the field of view image; periodically capture subsequent field of view images to track the screen over time; stop the screen timer when the screen can no longer be tracked; and record a screen time of a second instance based on the screen timer.
[0011] In some embodiments, the technology described herein relates to a head-mounted device, further comprising: at least one orientation sensor configured to detect orientation data of the head-mounted device; and at least one position sensor configured to detect relative position data of the device, wherein the processor is further configured to detect that the mode of the head-mounted device is a camera-free mode for screen time tracking; in the camera-free mode, the processor is configured to: capture the relative position data of the device and the orientation data of the head-mounted device; analyze the relative position data and the orientation data to identify a viewing condition; start a screen timer after the viewing condition is identified; periodically capture subsequent orientation data and subsequent relative position data to track the viewing condition over time; stop the screen timer when the viewing condition is no longer tracked; and record the screen time of a third instance based on the screen timer.
[0012] In some embodiments, the technology described herein relates to a head-mounted device, wherein at least one orientation sensor comprises an inertial measurement unit (IMU); and at least one position sensor comprises an ultra-wideband (UWB) sensor.
[0013] In some embodiments, the technology described herein relates to a head-mounted device, wherein a processor is further configured to: generate a notification based on screen time of a first instance, screen time of a second instance and / or screen time of a third instance; and display the notification on a display of the head-mounted device.
[0014] In some embodiments, the technology described herein relates to a computer-implemented method comprising: detecting that the mode of a head-mounted device for screen time tracking is a low-power mode; capturing an eye image of an eye using an eye-tracking camera of the head-mounted device; analyzing the eye image to identify a screen reflection; starting a screen timer after a screen reflection is identified; periodically capturing subsequent eye images to track the screen reflection over time; and stopping the screen timer when the screen reflection can no longer be tracked.
[0015] In some embodiments, the technology described herein relates to a computer-implemented method, wherein the step of periodically capturing subsequent eye images to track screen reflections over time includes the step of repeating the capture and analysis in a periodic period of more than 2 seconds.
[0016] In some embodiments, the technology described herein relates to a computer-implemented method, wherein the step of detecting that the mode of a head-mounted device for screen time tracking is a low-power mode comprises: detecting the battery level of the head-mounted device; and determining that the battery level is below a threshold value.
[0017] In some embodiments, the technology described herein relates to a computer-implemented method, wherein the step of analyzing an eye image to identify a screen reflection comprises: applying the eye image to the input of a neural network—the eye image includes an eye and a screen reflection—; and receiving a screen image at the output of the neural network—the screen image includes a screen reflection.
[0018] In some embodiments, the technology described herein relates to a computer-implemented method, further comprising: detecting that the mode of a head-mounted device for screen time tracking is a high-power mode; capturing a field of view image using a front camera of the head-mounted device; analyzing the field of view image to identify a screen; starting a screen timer after the screen is identified; periodically capturing subsequent field of view images to track the screen over time; and stopping the screen timer when the screen can no longer be tracked.
[0019] In some embodiments, the technology described herein relates to a computer-implemented method, wherein the step of detecting that the mode of a head-mounted device for screen time tracking is a high-power mode comprises: detecting the battery level of the head-mounted device; and determining that the battery level is exceeded a threshold.
[0020] In some embodiments, the technology described herein relates to a computer-implemented method, comprising: detecting that the mode of a head-mounted device for screen time tracking is a camera-free mode; capturing orientation data of a head-mounted device using at least one orientation sensor of the head-mounted device; capturing relative position data of the device from an ultra-wideband (UWB) signal received from the head-mounted device; analyzing relative position data and orientation data to determine a viewing condition; starting a screen timer after the viewing condition is identified; periodically capturing subsequent orientation data and relative position data to track the viewing condition over time; and stopping the screen timer when the viewing condition can no longer be tracked.
[0021] In some embodiments, the technology described herein relates to a computer-implemented method, further comprising the steps of: recording screen time in a database based on a screen timer; and generating a notification when screen time satisfies a criterion.
[0022] In some embodiments, the technology described herein relates to a computer-implemented method, further comprising the step of classifying instances of screen time corresponding to a device based on device information captured by a head-mounted device.
[0023] In some embodiments, the technology described herein relates to a head-mounted device comprising: an eye-tracking camera directed toward a user's eyes; a front camera directed toward a user's field of vision; at least one orientation sensor configured to detect orientation data of a head-mounted device worn by a user; and a processor, wherein the processor detects a low-power mode based on the battery level of the head-mounted device—in the low-power mode: screen time for the device is determined based on an eye image from the eye-tracking camera applied to a neural network—; and detects a high-power mode based on the battery level of the head-mounted device—in the high-power mode: screen time for the device is determined based on an image from the front camera applied to an image recognition algorithm—; and is configured by instructions.
[0024] In some embodiments, the technology described herein relates to a head-mounted device, wherein the processor is further configured by instructions for detecting a camera-free mode based on user input, and in the camera-free mode: screen time for the device is applied in a machine learning mode and is based on relative position data of the device and orientation data of the head-mounted device captured from an ultra-wideband (UWB) signal received from the head-mounted device.
[0025] In some embodiments, the technology described herein relates to a head-mounted device, the head-mounted device being augmented reality glasses (i.e., AR glasses).
[0026] In addition to the exemplary summary foregoing, other exemplary objects and / or advantages of the present disclosure and ways to achieve them are further described in the following detailed description and the accompanying drawings. Brief explanation of the drawing
[0027] FIG. 1 graphically illustrates screen time for a user over a predetermined period according to a possible implementation of the present disclosure. FIG. 2 is a perspective view of a head-mounted device according to a possible implementation of the present disclosure. FIG. 3 is a system block diagram of a head-mounted device according to a possible implementation of the present disclosure. FIG. 4 is a flowchart illustrating a multi-mode screen time tracking method according to a possible implementation of the present disclosure. Figure 5 is a block diagram of a possible neural network for the low-power mode of the multi-mode screen time tracking method illustrated in Figure 4. FIG. 6 illustrates a screen image identified in a reflection of an eye image according to a possible embodiment of the present disclosure. FIG. 7 is a flowchart of a method for recording screen time according to a possible implementation of the present disclosure. FIG. 8 is a drawing illustrating a notification generated based on screen time according to a possible implementation of the present disclosure. Components within a drawing do not necessarily need to be scaled relative to one another. Similar drawing symbols designate corresponding parts across multiple drawings. Specific details for implementing the invention
[0028] While it may be desirable to measure a person's total screen time over a set period, accurately measuring that total may not be practical when a person uses multiple devices. For example, a person might watch movies on television, play video games on a computer, read books on a tablet, and video chat on a mobile phone during the same day. Although one or more of these devices could be configured to record (i.e., measure) screen time, there is no way to measure and aggregate screen time across multiple devices for various reasons. For instance, some devices may not be able to easily exchange screen time information due to different operating systems, while others may not be able to measure or communicate this information at all (e.g., older televisions).
[0029] The present disclosure addresses the technical problem of measuring and recording (i.e., tracking) screen time across multiple devices through the use of a head-mounted device configured to monitor a user to determine the time a user spends engaging with the device's screen and to calculate the device's screen time based on this time. The head-mounted device may be further configured to track multiple screen times for each of the multiple devices. Additionally, the head-mounted device may be configured to combine the screen times of the devices to measure a person's total screen time. Additionally, the head-mounted device may be configured to take action when the screen time of one or more devices meets a criterion. For example, the criterion may be a threshold related to a maximum screen time over a given period, and the action may be related to a user's screen time limit. The criterion and period are configurable and may be suitable for various applications.
[0030] Screen time tracking may require periodic measurements using a head-mounted device. Periodic measurements can consume power, which may limit the usage time of the head-mounted device. For example, augmented reality glasses (i.e., AR glasses) may have limited power budgets due to their smaller battery sizes. Therefore, periodic screen time tracking can cause technical issues where the head-mounted device consumes more power than is actually required. The present disclosure further addresses the technical issues of power consumption associated with screen time tracking by disclosing a multi-mode approach having different power consumption for each mode. The screen time tracking mode can be selected based on the power status of the head-mounted device. For example, in a low battery state, a screen time tracking mode that consumes less power can be selected to extend the operating time of the head-mounted device. Additionally, the multi-mode approach may allow for additional modes that can provide additional benefits to the user. For example, a non-imaging mode for tracking screen time may be disclosed to enhance user privacy.
[0031] Users may not be aware of their accumulated screen time for a certain period. For example, users may become so distracted by screen content that they do not realize how long they have been observing the screen. The present disclosure further explains the need to notify users of screen time. For example, when the total screen time across a device reaches a certain amount, the head-mounted device may be configured to generate a notification and / or temporarily restrict the device's functions. These actions may help prevent or alleviate symptoms associated with excessive device use and help users comply with screen time rules. Additionally, these actions may help users or those responsible for them (e.g., parents, administrators) understand their behavior regarding screen time in general. For example, screen time tracking may be used to verify engagement (e.g., tracking user activities).
[0032] FIG. 1 graphically illustrates screen time for multiple devices. The screen time for each device is plotted on a timeline over a predetermined period (e.g., 24 hours). Screen time is plotted as a line for each instance of screen participation, and the length of each line corresponds to the screen time of the device. For example, an instance (110) of screen time for a desktop computer is plotted on the first graph (101). The screen time of the instance (110) can be calculated as the time elapsed between the start (111) (i.e., START) of the instance (110) and the end (112) (i.e., STOP) of the instance (110). The accumulated screen time for all devices (i.e., across devices) is also plotted as the total screen time over the period (100). The graph is provided for illustrative purposes. It should be understood that the depicted devices and screen times are fictitious and represent a possible implementation of one of many possible variations. For example, screen time for additional and / or different devices may be included.
[0033] As illustrated, the first graph (101) illustrates an instance of screen time corresponding to a first device (i.e., a desktop computer), the second graph (102) illustrates an instance of screen time corresponding to a second device (i.e., a television), the third graph (103) illustrates an instance of screen time associated with a third device (i.e., a mobile phone), and the fourth graph (104) illustrates an instance of screen time associated with a fourth device (i.e., a tablet). The fifth graph (105) illustrates an instance of screen time for all devices (i.e., across all devices). The instances of screen time can be summed to calculate the total screen time over a predetermined period.
[0034] Total screen time can be calculated in various ways. In a possible implementation, total screen time can be calculated as the cumulative sum of accumulated screen time. In another possible implementation, total screen time can be the sum of all instances of screen time measured over a given period. Thus, screen time can be measured as hours (e.g., minutes) or as a rate (e.g., minutes / period).
[0035] In a possible implementation, measuring screen time may include a screen timer. For example, the screen timer may be started and stopped for each instance of screen time during a specified period. Afterward, the screen timer may be reset when the period ends and restarted at the start of a subsequent period. The period (100) may be configured by a factory set or a user (e.g., a parent) based on requirements (e.g., health guidelines). The period (100) may also be adjustable. For example, the period may end when the total screen time is reached. For example, screen time may be accumulated until a threshold is reached, at which point the period ends.
[0036] Total screen time tracking can be customized to calculate a cumulative total of screen time based on the device, time period (e.g., time of day), and / or location. For example, screen time for a user's work device can be tracked at a time corresponding to a workday, and in possible implementations, daily, weekly, or monthly total screen time for the work device can be calculated. Additionally, a time period corresponding to specific screen time tracking can be triggered based on the location of the head-mounted device (i.e., the user). For example, a time period corresponding to a workday can begin when the user arrives at a location corresponding to work. The user is based on the device, time, and / or location.
[0037] FIG. 2 is a perspective view of a head-mounted device according to a possible embodiment of the present disclosure. As illustrated, the head-mounted device (200) may be implemented as smart glasses (e.g., AR glasses) configured to be worn on a user's head. The head-mounted device (200) includes a left lens and a right lens, each of which is attached to the user's ears by a left arm and a right arm. The user (i.e., the wearer) can see the world through the left lens and the right lens, which are joined together by a bridge configured to rest over the wearer's nose.
[0038] The head-mounted device (200) may include a sensing device configured to determine the position where the user's focus is directed. For example, the head-mounted device (200) may include at least one front-facing camera. The front-facing camera (230) may include an image sensor capable of detecting intensity levels of visible light and / or near-infrared light (i.e., NIR light). The front-facing camera (230) may be directed toward a forward field of view (i.e., a forward FOV (235)) or may include an optical device that routes light from the forward FOV (235) to the image sensor. For example, the front-facing camera (230) may be placed on the front surface of the head-mounted device and may include at least one lens for generating an image of the forward field of view (forward FOV (235)) on the image sensor. The front FOV (235) may include all (or part) of the user's field of view, so that an image or video of the world from the user's point of view (POV) can be captured by the front camera (230).
[0039] The head-mounted device (200) may further include at least one eye-tracking camera. The eye-tracking camera (220) may be directed toward the eye's field of view (i.e., eye-FOV (225)) or may include an optical device that routes light from the eye-FOV (225) to an eye image sensor. For example, the eye-tracking camera (220) may be directed toward the user's eye and may include at least one lens for generating an image of the eye-FOV (225) on the eye image sensor. The eye-FOV (225) may include all (or part) of the user's field of view so that an image or video of the eye is generated. The image of the eye may be analyzed by a processor of the head-mounted device (not shown) to determine the position the user is looking at. For example, the relative position of the pupil in the image of the eye may correspond to the direction of the user's gaze.
[0040] The head-mounted device may further include at least one orientation sensor (250). The orientation sensor(s) may be implemented as a combination of an accelerometer, a gyroscope, and a magnetometer combined to form an inertial measurement unit (i.e., an IMU) to determine the orientation of the head-mounted device. The IMU may be configured to provide multiple measurements describing the orientation and motion of the head-mounted display. For example, the IMU may have six degrees of freedom (6-DOF) capable of describing three translational movements (i.e., x direction, y direction, or z direction) along the axes of the world coordinate system (260) and three rotational movements (i.e., pitch, yaw, and roll) about the axes of the world coordinate system (260). Data from the IMU may be combined with information regarding the Earth's magnetic field using sensor fusion to determine the orientation of the head-mounted device coordinate system (270) relative to the world coordinate system (260). Information from the front camera (230), eye-FOV (225), and IMU (250) can be combined to determine the position where the user's focus is directed, which can enable augmented reality applications. The head-mounted display may also include an interface device for these applications.
[0041] The head-mounted device (200) may further include a human interface subsystem configured to provide information to a user. For example, the head-mounted device (200) may include a display configured to display information (e.g., text, graphics, images) in a display area (240) of one or both lenses. The display area may be all or part of the lens and may be visually transparent or translucent so that the user can see through the display area when not in use. The head-mounted device (200) may further include one or more speakers (e.g., earbuds (255)) configured to play sound (e.g., voice, music, tone). The present disclosure describes a system and method for configuring a sensing device and an interface device of the head-mounted device for screen time measurement and response (e.g., notification).
[0042] FIG. 3 is a block diagram of a head-mounted device according to a possible implementation of the present disclosure. As described above, the head-mounted device (200) includes a front camera (230) configured to face the user's viewpoint and capture an image of the forward FOV (235). The head-mounted device (200) further includes an eye-tracking camera (220) configured to face the user's eyes (or both eyes) and capture an image of the eye FOV (225). The head-mounted device (200) further includes at least one orientation sensor (250) (e.g., an IMU) configured to detect the movement, position, and / or orientation of the head of a person wearing the head-mounted device (200).
[0043] The head-mounted device (200) may further include at least one position sensor (310). At least one position sensor may be configured to determine the position of the head-mounted device (i.e., the user). The position sensor (310) may include an ultra-wideband (UWB) sensor. The position sensor (310) may communicate with the screen device (317) via a communication link (315). For example, the head-mounted device (200) and the screen device (317) may exchange packets of information via the UWB communication link to determine the relative position of the devices. For example, the position sensor may be configured to determine the round-trip time (RTT) of the packets to be communicated between the devices. The range between the head-mounted device and the screen device (317) may be detected based on the round-trip time. Additionally, the position sensor may include a plurality of receivers configured to receive the packets communicated from the screen device (317). The position sensor may be configured to determine the time at which a packet arrives at the receiver to determine the angle between the screen device (317) and the position sensor (310). The position sensor(s) may further include a global positioning system (GPS) sensor that can be used to determine the geographic location of the head-mounted device (i.e., the user). The geographic location may be further determined through a sensor fusion approach in which information from a local area network (e.g., a WiFi network) and / or a cellular network can further specify the geographic location.
[0044] The head-mounted device (200) further includes at least one processor (330). The processor communicates with the camera, sensors, and other modules and electronics of the head-mounted device. The processor is configured by instructions (e.g., software, applications, etc.) to perform a multi-mode method for tracking screen time across the device. The instructions may be non-transient computer-readable instructions stored in memory (340) and called from memory. Alternatively, the instructions may be communicated to the processor from a computer connected to the network (370) via a communication interface (350).
[0045] The processor (330) can communicate with the battery (360). The processor (330) can be configured by instructions to monitor the battery level and determine the mode of the head-mounted device based on the battery level. For example, when the battery level is below a threshold, the head-mounted device may be determined to be in a low-power mode, and when the battery level exceeds the threshold, the head-mounted device may be determined to be in a high-power mode.
[0046] The processor can communicate with the display (320) of the head-mounted device (200). The processor (330) can be configured by commands to transmit text, graphics, video, images, etc. to the display (320). For example, the processor (330) can be configured to display a notification to the user on the display (320) in response to measured screen time.
[0047] FIG. 4 is a flowchart of a multi-mode screen time tracking method according to a possible implementation of the present disclosure. The method (400) includes a step (410) of detecting a mode (i.e., screen time mode) to be used by a head-mounted device to measure screen time. The step of detecting a screen time mode (i.e., mode) can be implemented in various ways. In one possible implementation, the step of detecting a mode (410) may include receiving input from a user specifying a mode (i.e., user input (402)). In another possible implementation, the step of detecting a mode (410) may include receiving a mode from a device setting (403) of a head-mounted device, and the device setting (i.e., system setting) may be configured by an application running on the head-mounted device (e.g., an AR application). In another possible implementation, the step of detecting a mode (410) may include determining a mode based on the battery level (401) of the head-mounted device in an effort to balance the quality of tracking with the operation time of the head-mounted device.
[0048] In a possible implementation, the detected mode may be a high-power mode (490), a low-power mode (600), or a camera-free mode (900). Each mode may have different accuracy and power consumption. For example, the head-mounted device (200) may consume more power when in the high-power mode (490) than when in the low-power mode (600). The high-power mode (490) may generate screen time with higher accuracy (i.e., reliability) than the low-power mode (600). The camera-free mode (900) may be used without any imaging, which may provide privacy to the user and provide the lowest power consumption among the modes. However, the camera-free mode (900) may be optional because it may rely on position and orientation sensors that may or may not be present in all screen devices.
[0049] In a possible implementation, screen time tracking can be used alternately between modes as needed for accuracy or battery life. For example, screen time tracking for a device can be started in high-power mode (490) and then switched to low-power mode (600) between instances or within the same instance. This allows for recognizing the device or screen content with higher reliability at the start of the screen time instance before switching to low-power mode, which can help track screen time more efficiently. In another example, high-power mode can be triggered when a screen is detected in low-power mode. This can help improve accuracy only when it is determined with some certainty that the screen is being viewed.
[0050] As illustrated in the implementation of FIG. 4, the high-power mode (490) uses the front camera (230) of the head-mounted device and an image recognition algorithm (424) to determine the start (i.e., START) and end (i.e., STOP) of an instance of screen time. The low-power mode (600) uses the eye-tracking camera (220) of the head-mounted device and a neural network (434) (e.g., U-Net neural network) to determine the start / end of an instance of screen time. The camera-free mode (900) uses the position / orientation sensor (443) of the head-mounted device and a machine learning model (e.g., sensor fusion) to determine the start / end of an instance of screen time.
[0051] Each mode can trigger the screen timer (430) to turn ON / OFF according to the instance's START / STOP. That is, the screen timer (430) can be used to determine screen time regardless of the mode used to determine the instance's START / STOP of screen time. For example, the screen timer can be activated (i.e., started) when it is determined that the screen is in the user's line of sight, and then deactivated (i.e., stopped) when it is determined that the screen is no longer in the user's line of sight. The determination of screen viewing conditions can be performed differently depending on the mode. The screen timer (430) can be reset when the instance ends. Alternatively, the screen timer (430) can be turned ON / OFF without reset to accumulate screen time for multiple instances over a predetermined period, and then reset at the end of the predetermined period.
[0052] As illustrated in FIG. 4, the method (400) may include a high-power mode (490). The high-power mode (490) uses a front camera (230) of a head-mounted device that consumes more power than the eye-tracking camera (220). In a low-power mode (600), the front camera (230) may be configured to ON (i.e., enabled, powered on, etc.) to capture an image or image corresponding to the user's field of vision. For example, the front camera (230) captures a video (e.g., high-resolution, color video) of an item in the user's field of vision. The image (or image) can be analyzed by an image recognition algorithm (424) to identify (i.e., detect) the screen(s) that the user is viewing. Detection of the screen can trigger a screen timer (430) to start timing.
[0053] The aforementioned high-power mode (490) process may be repeated periodically (i.e., cycled) to track the screen in subsequently captured images to determine the screen time of the instance. The step of determining the screen time of the instance may include determining when the screen timer (430) should be stopped (i.e., disabled, turned off). For example, if it is determined that no screen is detected in the images for a number of cycles, the screen timer may be turned OFF. Each cycle may be separated into cycle periods. For example, the high-power mode (490) may be repeated every few seconds (e.g., 2 seconds) in cycle periods. When the instance is terminated, the screen timer may output the screen time of the instance for recording and / or monitoring.
[0054] As illustrated in FIG. 4, the method (400) may include a low-power mode (600). The low-power mode (600) uses an eye-tracking camera (220) of a head-mounted device that consumes less power than a front camera. In the low-power mode (600), the eye-tracking camera (220) may be configured to ON (i.e., enabled, powered on, etc.) to capture an image or image corresponding to the user's eye (or eyes). For example, the eye-tracking camera (220) may capture a video of the user's eye. The image of the eye may be analyzed by a neural network to detect reflections and identify the reflected screen. Identification of the screen may trigger a screen timer (430) to start timing.
[0055] FIG. 5 is a flowchart block diagram of a possible neural network for a low-power mode of the multi-mode screen time tracking method illustrated in FIG. 4. The neural network is a U-net neural network (500) configured to receive an eye image (510) (i.e., input image) from an eye tracking camera (220). The U-net neural network (500) may be configured to detect reflections in the eye image and identify the reflected screen. Identification may include the steps of generating a mask containing the reflection and applying the mask to the eye image (510) of the eye to generate a screen image (520) (i.e., output image). That is, the U-net neural network may be configured to predict whether a screen appears in the eye image, and if it appears, to predict where the screen is located in the eye image (i.e., the center position of the screen).
[0056] FIG. 6 illustrates a screen image identified from a reflection within an eye image according to a possible embodiment of the present disclosure. As illustrated, a reflection may be detected in the eye image (610). A screen may be identified from the reflection by a neural network to generate a screen image (620). In this possible embodiment, the screen is a total reflection. That is, the screen image (620) may be generated by masking (i.e., segmenting) the eye image (610) based on the result of a U-net neural network (500) (see FIG. 5) configured to position the screen in the image.
[0057] Returning to FIG. 4, the aforementioned high-power mode (600) process may be repeated periodically (i.e., cycled) to track the screen in subsequently captured images to determine the screen time for instances of screen viewing. The step of determining the screen time for an instance may include determining when the screen timer (430) should be stopped (i.e., disabled, turned off). For example, if it is determined that no screen is detected in the images for a number of cycles, the screen timer may be turned OFF. Each cycle may be separated into cycle periods. For example, the low-power mode may be repeated every few seconds (e.g., 5 seconds) in cycle periods. The cycle period of the low-power mode may be longer than the cycle period of the high-power mode to conserve power. The cycle period may be selected as a balance between screen time accuracy and power consumption. In a possible implementation, the low-power mode (600) may be the default mode of the method.
[0058] As illustrated in FIG. 4, the method (400) may include a camera-free mode (900). The camera-free mode (900) uses a position / orientation sensor (443) of a head-mounted device to determine the user's orientation data and the relative position data of the device. The user's orientation data and the relative position data of the device (i.e., screen) can be analyzed by a machine learning model (444) to determine whether a screen viewing condition (i.e., a viewing condition) exists. For example, if the orientation of the head-mounted device faces the direction of the device having the screen, a viewing condition may exist. When a viewing condition is detected, a screen timer (430) may be triggered to start timing. As described above, the relative position data may include a range and an angle calculated based on the use of ultra-wideband (UWB) communication. For example, the range between UWB devices may be calculated based on the round-trip time of UWB packets communicating between UWB devices. In addition, the angle between UWB devices can be calculated based on the difference in arrival times of UWB packets at spaced-apart UWB receivers on the UWB device receiving the UWB packet.
[0059] The aforementioned no-camera mode (900) process may be repeated periodically (i.e., cycled) to track viewing conditions in order to determine screen time for instances of screen viewing. The step of determining screen time for an instance may include determining when the screen timer (430) should be stopped (i.e., disabled, turned off). For example, if it is determined that no viewing conditions are detected in the position / orientation data for a number of cycles, the screen timer (430) may be turned OFF. Each cycle may be separated into cycle periods. For example, the low-power mode may be repeated every few seconds (e.g., 5 seconds) in cycle periods.
[0060] As illustrated in FIG. 4, after the screen time is calculated, it can be output for the recording / monitoring process (450) of the method (400). The recording / monitoring process (450) may include the step of recording the screen time determined from the screen timer into the database (750). The recording / monitoring process (450) may further include the step of linking the screen time (455) to the device (i.e., classifying the screen time as being for the device). The recording / monitoring process (450) may further include the step (465) of generating a notification based on the screen time or screen times stored in the database (750). For example, the screen time for an instance may be output from the screen timer in high-power mode, low-power mode, or no-camera mode and classified, recorded, and evaluated to generate a notification.
[0061] FIG. 7 is a flowchart of a method for recording screen time according to a possible implementation of the present disclosure. The method (700) includes the step of receiving screen time (e.g., of an instance) acquired using a head-mounted device in a high-power mode, a low-power mode, or a camera-free mode. The method (700) further includes the step (710) of classifying screen time, such as screen time corresponding to the device.
[0062] The classification step (710) may utilize a machine learning model (715). The machine learning model (715) may receive device information (720) acquired during the instance to link screen time (701) to a class of devices (e.g., laptop, desktop, television, etc.) or a specific device (e.g., John's phone, living room TV, etc.). Device information may affect the reliability of the classification. For example, if a hand holding a screen is recognized in an image, the model may then lower the reliability that the screen is a television and raise the reliability that the screen is a mobile phone or tablet.
[0063] Device information (720) may include screen characteristics (721) derived from a captured image (e.g., FOV image). In a possible implementation, the screen characteristics (721) may include physical or electronic properties of the screen derived from the image, such as screen size (e.g., length, width), screen resolution, or screen refresh rate. In another possible implementation, the screen characteristics (721) may include properties of the surrounding environment of the screen derived from the image, such as the housing of the screen or a hand holding the screen.
[0064] The device information (720) may further include screen content (722) derived from a captured image (e.g., an eye image). In a possible implementation, the screen content may include motion, color, image, etc., that suggest the type of screen being observed. For example, a specific user interface may be recognized and used to determine the device.
[0065] The device information (720) may further include a device identification (device ID (723)) derived from location data. In a possible implementation, the device ID (723) may include identification information (e.g., an address) communicated via a UWB packet. For example, an address communicated in a packet transmitted from a device may be used to determine the device.
[0066] The device information (720) may further include a user location (724). The user location may be a location derived from a location service (e.g., GPS, indoor positioning, etc.). The user location (724) may be a room. Knowledge of the room may inform a decision constructed by a machine learning model. For example, a TV is more likely to be in the living room than in the bathroom. That is, the method may include the step of receiving the location of a head-mounted device; and the step of selecting a possible device based on the location.
[0067] A method for recording screen time may further include a step (730) of verifying whether the classification configured by the machine learning model (715) was accurate or inaccurate. Verification may be performed in various ways. In one possible implementation, an image from the front camera may be captured (e.g., during low-power mode). The FOV image may be analyzed to determine whether the classification of the device was accurate. In another possible implementation, verification may include a step of querying the user to classify the device (e.g., if no result is returned from the model) or verifying whether the classification was accurate. The result of verification may be used to update the model (740). For example, the step of updating the model (740) may include a step of adjusting the model so that the confidence level for the classification resulting from specific device information (720) is changed based on instances of verification or verification rates (e.g., false identification rate, true identification rate).
[0068] A method for recording screen time may further include the step of recording (classified) screen time in a database (750). The recording step may include the step of updating (i.e., incrementing) the screen time for a device based on the screen time (701) received (or derived) from the screen timer. Accordingly, the recording step (745) may include the step of querying the database for an entry corresponding to the device, and then adding the screen time for the device to the entry or creating a new entry for the device.
[0069] Returning to FIG. 4, the multi-mode screen time tracking method (400) may further include the steps of generating a notification based on screen time recorded in a database (750) and transmitting the notification to a display of a head-mounted device. The step of generating a notification may include the step of querying the database based on one or more rules corresponding to one or more notifications. For example, there may be a rule that the screen time of a specific device must not exceed a certain value for a certain period. The step of generating a notification may include the step of querying the database (750) to determine the screen time for a specific device and then comparing the screen time with a threshold. If the comparison indicates that the screen time for a specific device is above the threshold, then a notification may be generated. Various notifications may be generated.
[0070] Notifications may include messages provided to the user regarding screen time. For example, the message may instruct the user to take a "break" from using the device. Notifications may include reports on screen time usage for a specified period (or periods). Notifications may include comparing a first screen time of a first period with a second screen time of a second period. These types of notifications may be displayed on the display of a head-mounted device so that the user can observe them.
[0071] Figure 8 is an example of a notification based on screen time. The notification may include daily screen time for all devices (e.g., desktop, mobile phone, tablet, etc.) plotted as a bar chart by day of the week. The notification may further include accumulated screen time for each device over a week. The notification may further include total screen time for all devices over a week. The notification may further include screen time analysis, such as daily average screen time. The notification may further include changes in screen time from a previous period (e.g., last week). The notification may be triggered to be displayed at the end of a period (e.g., week) or may be triggered by the user. In some implementations, the notification may be generated when one or more screen times satisfy one or more criteria.
[0072] In the specification and / or drawings, typical embodiments are disclosed. The present disclosure is not limited to these exemplary embodiments. The use of the term "and / or" includes any and all combinations of one or more of the related listed items. As the drawings are schematic representations, they are not necessarily drawn to scale. Unless otherwise stated, specific terms are used in a general and descriptive sense, not for the purpose of limitation.
[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art. Methods and materials similar or equivalent to those described herein may be used in the practice or testing of this disclosure. As used in the specification and appended claims, the singular forms (“a,” “an,” “the”) include the plural forms unless otherwise indicated in the context. The term “comprising” and its variations used herein are used synonymously with the term “including” and its variations and are open and non-restrictive terms. The terms “optional” or “optional” used in this document mean that the features, events, or situations described below may or may not occur, and that the description includes cases where said features, events, or situations occur and cases where they do not occur. A scope may be expressed herein as one specific value “approximately” and / or another specific value “approximately”. Where such a scope is expressed, the embodiment includes one specific value and / or another specific value. Likewise, it will be understood that when a value is expressed as an approximation using the antecedent "approximately," a specific value forms another modality. It will also be understood that each endpoint of a range is important both in relation to other endpoints and independently of other endpoints.
[0074] Some implementations may be realized using various semiconductor processing and / or packaging technologies. Some implementations may be realized using various types of semiconductor processing technologies associated with semiconductor substrates, including, but not limited to, silicon (Si), gallium arsenide (GaAs), gallium nitride (GaN), silicon carbide (SiC), etc.
[0075] Although specific features of the described implementation have been illustrated as described herein, those skilled in the art may now consider many modifications, substitutions, changes, and equivalents. Accordingly, it should be understood that the appended claims are intended to cover all modifications and changes that fall within the scope of the implementation. It should be understood that this is presented as an example and is not limited thereto, and that the form or details may vary. Any part of the device and / or method described herein may be combined in any combination, except mutually exclusive combinations. The implementation described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the various implementations described.
[0076] In the foregoing description, where it is stated that an element is upon, connected to, electrically connected to, coupled to, or electrically coupled to another element, it will be understood that this means the element is directly upon, connected to, or coupled to the other element, or that one or more intervening elements may be present. Conversely, where it is stated that an element is directly upon or directly connected to another element, no intervening elements are present. Throughout the detailed description, the terms "directly upon" or "directly connected" may not be used, but elements indicated as being directly upon or directly connected may be referred to as such. The claims of the application may be modified to refer to exemplary relationships described in the specification or illustrated in the drawings.
[0077] In this specification, the singular form may include the plural form unless the context specifies a special case. Spatially relative terms (e.g., “above,” “upper,” “upper,” “lower,” “under,” “below,” “lower,” etc.) are intended to include different orientations of the device when used or operated in addition to the orientations illustrated in the drawings. In some implementations, the relative terms above and below may include vertical above and vertical below, respectively. In some implementations, the term adjacent may include laterally adjacent or horizontally adjacent.
Claims
Claim 1 A computer-implemented method comprising: capturing an eye image using an eye-tracking camera of a head-mounted device, wherein the eye image is an image of the surface of the eye of a user wearing the head-mounted device; analyzing the eye image to identify a screen reflection, wherein the screen reflection corresponds to a screen positioned in front of the user and reflected on the surface of the user's eye; starting a screen timer after the screen reflection is identified; tracking the screen reflection over time; and stopping the screen timer when the screen reflection is no longer identified. Claim 2 A computer-implemented method according to claim 1, wherein the step of tracking the screen reflection over time includes the step of repeating capture and analysis at regular intervals to periodically capture subsequent eye images to track the screen reflection over time. Claim 3 A computer-implemented method according to claim 1, further comprising the step of detecting that the head-mounted device is in a low-power mode, wherein the step of detecting the low-power mode comprises: the step of detecting the battery level of the head-mounted device; and the step of determining that the battery level is less than a threshold value. Claim 4 A computer-implemented method according to claim 1, wherein the step of analyzing the eye image to identify the screen reflection comprises: applying the eye image to the input of a neural network, wherein the eye image includes the eye and the screen reflection; and receiving the screen image at the output of the neural network, wherein the screen image includes the screen reflection. Claim 5 A computer-implemented method according to claim 1, further comprising: detecting that the head-mounted device is in a high-power mode for tracking screen time; capturing a field of view image using a front camera of the head-mounted device; analyzing the field of view image to identify the screen; starting the screen timer after the screen is identified; periodically capturing subsequent field of view images to track the screen over time; and stopping the screen timer when the screen is not identified in the subsequent field of view images. Claim 6 In claim 5, the step of detecting that the head-mounted device is in a high-power mode for tracking screen time comprises: detecting the battery level of the head-mounted device; and determining that the battery level is exceeded a threshold value, a computer-implemented method. Claim 7 A computer-implemented method according to claim 5, further comprising: detecting that the head-mounted device is in a camera-free mode for screen time tracking; capturing orientation data of the head-mounted device using at least one orientation sensor of the head-mounted device; capturing relative position data of the device from an ultra-wideband (UWB) signal received from the head-mounted device; analyzing the relative position data and the orientation data to determine a viewing condition; starting a screen timer after the viewing condition is identified; periodically capturing subsequent orientation data and relative position data to track the viewing condition over time; and stopping the screen timer when the viewing condition can no longer be tracked. Claim 8 A computer-implemented method according to claim 7, further comprising the step of recording screen time in a database based on the screen timer; and the step of generating a notification when the screen time satisfies a criterion. Claim 9 A computer-implemented method according to claim 8, further comprising the step of classifying instances of screen time corresponding to a device based on device information captured by the head-mounted device. Claim 10 A computer-implemented method according to paragraph 2, wherein the cycle period is longer than 2 seconds. Claim 11 A computer-implemented method according to paragraph 2, wherein the cycle period is shorter than 2 seconds. Claim 12 A head-mounted device comprising: an eye-tracking camera; and a processor communicating with said eye-tracking camera, wherein the processor is configured to perform the method described in any one of claims 1 to 11 by means of software instructions. Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete Claim 21 delete
Citation Information
Patent Citations
Multi-mode eye tracking
KR1020190138852A
User recognition and gaze tracking in a video system
US20210201021A1
Systems and methods for digital wellness
WO2021212227A1
