Head-mounted device for tracking screen time
Through a multi-mode head-mounted device combining eye tracking cameras, front cameras and sensors to identify screen reflection and viewing conditions, the problem of measuring screen time across multiple devices is solved, reducing power consumption, extending device usage time, and through Alarm reminds the user to control screen time.
Patent Information
- Application Number
- CN202380038201.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-16
- Filing Date
- 2023-05-05
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to effectively measure and record screen time across multiple devices, and regular screen time tracking consumes high power, limiting the use time of the head-mounted device.
Design a multi-mode head-mounted device that uses eye tracking cameras, front cameras and sensors to identify screen reflection and viewing conditions through neural networks and machine learning models to achieve screen time tracking in low-power and high-power modes, and in In camera-free mode, track viewing conditions through position and orientation sensors, selecting the appropriate mode to reduce power consumption.
Accurate screen time measurement and recording across multiple devices is achieved, extends the operation time of the head-mounted device, provides additional privacy mode, and alerts users to control screen time, preventing symptoms related to overuse.
Smart Images

Figure CN119072676A_ABST
Abstract
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of, and claims priority to, U.S. Application No. 17 / 663,444, filed on May 16, 2022. The disclosure of the U.S. Application is incorporated herein by reference in its entirety. Technical Field
[0002] The present disclosure relates to head mounted devices, and more particularly, to head mounted devices configured to measure screen time across multiple devices. Background Art
[0003] Screen time is the amount of time a person spends using a device with a screen. Excessive screen time can negatively impact sleep, physical health, brain development, and / or behavior. Summary of the invention
[0004] In some aspects, the technology described herein relates to a head-mounted device including: an eye tracking camera directed toward an eye of a user; and a processor in communication with the eye tracking camera, the processor being configured by instructions to: capture an eye image of the eye of the user using the eye tracking camera; analyze the eye image to identify a screen reflection; start a screen timer after identifying the screen reflection; periodically capture subsequent eye images to track the screen reflection over time; stop the screen timer when the screen reflection can no longer be tracked; and record the screen time of the first instance based on the screen timer.
[0005] In some aspects, the technology described herein relates to a head-mounted device in which analyzing eye images to identify screen reflections is performed by a neural network.
[0006] In some aspects, the technology described herein relates to a head-mounted device, wherein the processor is further configured to: classify screen time of a first instance as for a first device based on a machine learning model configured with device information captured by the head-mounted device; and record the screen time of the first instance for the first device in a database.
[0007] In some aspects, the technology described herein relates to a head-mounted device, wherein the device information is any combination of a device identifier, content reflected by the screen, characteristics of the screen reflection, and a location of the user.
[0008] In some aspects, the technology described herein relates to a head-mounted device, wherein the processor is further configured by instructions to: detect that a mode of the head-mounted device for screen time tracking is a low power mode.
[0009] In some aspects, the technology described herein relates to a head-mounted device, further comprising: a front-facing camera, and wherein: the processor is further configured to: detect that the mode of the head-mounted device for screen time tracking is a high-power mode: capture a field of view image of a user's field of view using the front-facing camera; analyze the field of view image to identify a screen; start a screen timer after identifying the screen in the field of view image; periodically capture subsequent field of view images to track the screen over time; stop the screen timer when the screen can no longer be tracked; and record the screen time of a second instance based on the screen timer.
[0010] In some aspects, the technology described herein relates to a head-mounted device, further comprising: at least one orientation sensor configured to sense orientation data of the head-mounted device; and at least one position sensor configured to sense relative position data of the device, and wherein: the processor is further configured to: detect that the mode of the head-mounted device for screen time tracking is a camera-free mode for screen time tracking; and in the camera-free mode, the processor is configured to: capture orientation data of the head-mounted device and relative position data of the device; analyze the orientation data and the relative position data to identify viewing conditions; after identifying the viewing conditions, start a screen timer; periodically capture subsequent orientation data and subsequent relative position data to track the viewing conditions over time; stop the screen timer when the viewing conditions can no longer be tracked; and record the screen time of a third instance based on the screen timer.
[0011] In some aspects, the technology described herein relates to a head-mounted device wherein: at least one orientation sensor comprises an inertial measurement unit (IMU); and at least one position sensor comprises an ultra-wideband (UWB) sensor.
[0012] In some aspects, the technology described herein relates to a head-mounted device, wherein the processor is further configured to: generate an alert based on the screen time of a first instance, the screen time of a second instance, and / or the screen time of a third instance; and display the alert on a display of the head-mounted device.
[0013] In some aspects, the technology described herein relates to a computer-implemented method comprising: detecting that a mode of a head-mounted device for screen time tracking is a low power mode; capturing an eye image of an eye using an eye tracking camera of the head-mounted device; analyzing the eye image to identify a screen reflection; starting a screen timer after identifying the screen reflection; periodically capturing subsequent eye images to track the screen reflection over time; and stopping the screen timer when the screen reflection can no longer be tracked.
[0014] In some aspects, the technology described herein relates to a computer-implemented method in which periodically capturing subsequent eye images to track screen reflections over time includes repeating the capturing and analyzing with a loop period greater than 2 seconds.
[0015] In some aspects, the technology described herein relates to a computer-implemented method in which detecting that a mode of a head-mounted device for screen time tracking is a low power mode includes: detecting a battery level of the head-mounted device; and determining that the battery level is below a threshold.
[0016] In some aspects, the technology described herein relates to a computer-implemented method in which analyzing an eye image to identify screen reflections includes: applying the eye image to an input of a neural network, the eye image including the eye and the screen reflection; and receiving the screen image at an output of the neural network, the screen image including the screen reflection.
[0017] In some aspects, the technology described herein relates to a computer-implemented method further comprising: detecting that a mode of a head-mounted device for screen time tracking is a high power mode; capturing a field of view image using a front-facing camera of the head-mounted device; analyzing the field of view image to identify a screen; starting a screen timer after identifying the screen; periodically capturing subsequent field of view images to track the screen over time; and stopping the screen timer when the screen can no longer be tracked.
[0018] In some aspects, the technology described herein relates to a computer-implemented method in which detecting that a mode of a head-mounted device for screen time tracking is a high power mode includes: detecting a battery level of the head-mounted device; and determining that the battery level is above a threshold.
[0019] In some aspects, the technology described herein relates to a computer-implemented method further comprising: detecting that a mode of a head-mounted device for screen time tracking is a no-camera mode; capturing orientation data of the head-mounted device using at least one orientation sensor of the head-mounted device; capturing relative position data of the device from an ultra-wideband (UWB) signal received at the head-mounted device; analyzing the orientation data and the relative position data to determine viewing conditions; upon identifying the viewing conditions, starting a screen timer; periodically capturing subsequent orientation data and relative position data to track the viewing conditions over time; and stopping the screen timer when the viewing conditions can no longer be tracked.
[0020] In some aspects, the technology described herein relates to a computer-implemented method further comprising: recording screen time in a database based on a screen timer; and generating an alert when the screen time meets a criterion.
[0021] In some aspects, the techniques described herein relate to a computer-implemented method further comprising: classifying instances of screen time as corresponding to a device based on device information captured by a head-mounted device.
[0022] In some aspects, the technology described herein relates to a head-mounted device comprising: an eye-tracking camera directed toward an eye of a user; a front-facing camera directed toward a field of view of the user; at least one orientation sensor configured to sense orientation data of the head-mounted device worn by the user; and a processor configured via instructions to: detect a low power mode based on a battery level of the head-mounted device, wherein in the low power mode: a screen time of the device is determined based on an eye image from the eye-tracking camera that is applied to a neural network; and detect a high power mode based on the battery level of the head-mounted device, wherein in the high power mode: a screen time of the device is determined based on an image from the front-facing camera that is applied to an image recognition algorithm.
[0023] In some aspects, the technology described herein relates to a head-mounted device, wherein the processor is further configured by instructions to: detect a no-camera mode based on user input, wherein in the no-camera mode: the screen time of the device is based on orientation data of the head-mounted device and relative position data of the device, the relative position data captured from an ultra-wideband (UWB) signal received at the head-mounted device and applied to a machine learning model.
[0024] In some aspects, the technology described herein relates to a head-mounted device, wherein the head-mounted device is augmented reality glasses (ie, AR glasses).
[0025] The foregoing exemplary inventive content and other exemplary purposes and / or advantages of the present disclosure and their implementation are further explained in the following detailed description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 According to a possible implementation of the present disclosure, the screen time of a user within a time period is graphically shown.
[0027] Figure 2 is a perspective view of a head mounted device according to a possible implementation of the present disclosure.
[0028] Figure 3 is a system block diagram of a head mounted device according to a possible implementation of the present disclosure.
[0029] Figure 4 is a flow chart illustrating a multi-modal screen time tracking method according to a possible implementation of the present disclosure.
[0030] Figure 5 yes Figure 4 Block diagram of a possible neural network for the low-power mode of the multi-modal screen time tracking method shown.
[0031] Figure 6 Shown is a screen image identified in a reflection in an eye image according to a possible implementation of the present disclosure.
[0032] Figure 7 A flowchart of a method for recording screen time according to a possible implementation of the present disclosure is shown.
[0033] Figure 8 is an alarm generated based on screen time according to a possible implementation of the present invention.
[0034] The components in the drawings are not necessarily drawn to scale relative to each other. Like reference numerals designate corresponding parts throughout the several views. DETAILED DESCRIPTION
[0035] It may be desirable to measure a person's total screen time over a period of time; however, when the person uses multiple devices, it may not be practical to accurately measure the person's total screen time. For example, over the course of a day, a person may watch a movie on a TV, play a video game on a computer, read a book on a tablet, and video chat on a mobile phone. Although one or more of these devices may be configured to record (i.e., measure) screen time, screen time cannot be measured and counted across multiple devices for various reasons. For example, some of the devices may not be able to easily exchange screen time information due to different operating systems, while other devices may not be able to measure or communicate this information at all (e.g., old TVs).
[0036] The present disclosure solves the technical problem of measuring and recording (i.e., tracking) screen time across multiple devices by using a head-mounted device, which is configured to monitor a user to determine the time when the user interacts with the screen of the device and calculate the screen time of the device based on these times. The head-mounted device can be further configured to track multiple screen times for each of the multiple devices. In addition, the head-mounted device can be configured to merge the screen time of the device to measure the total screen time of the person. In addition, the head-mounted device can be configured to take action when the screen time of one or more devices meets the criteria. For example, the criteria can be a threshold value related to the maximum screen time within a time period, and the action can be related to limiting the user's screen time. The criteria and time periods are configurable and can be suitable for various applications.
[0037] Screen time tracking may require periodic measurements using a head-mounted device. Periodic measurements may consume power, which may limit the usage time of the head-mounted device. For example, augmented reality glasses (i.e., AR glasses) may have a limited power budget due to the small battery size. Therefore, periodic screen time tracking may bring about the technical problem that the power consumed exceeds the actual power of the head-mounted device. The present disclosure further solves the power consumption technical problem associated with screen time measurement by disclosing a multi-mode method, in which each mode has a different power consumption. The mode of screen time tracking can be selected based on the power condition of the head-mounted device. For example, a screen time tracking mode that consumes less power can be selected in a low battery condition to extend the operating time of the head-mounted device. In addition, the multi-mode method can allow additional modes that may have additional benefits to the user. For example, a non-imaging mode for tracking screen time is disclosed, which can enhance the privacy of the user.
[0038] The user may not be aware of their accumulated screen time during the period. For example, the user may be distracted by the content of the screen so that they do not realize how long they have been looking at the screen. The present disclosure further addresses the need to remind users of screen time. For example, when the total screen time across devices reaches a certain amount, the head-mounted device can be configured to generate an alarm and / or temporarily limit the function of the device. These actions can help prevent or alleviate symptoms associated with excessive use of the device and can help users comply with rules about screen time. In addition, these actions can help the user or someone responsible for the user (e.g., a parent, a manager) to get a general understanding of their behavior regarding screen time. For example, screen time tracking can be used to verify interaction (e.g., tracking the user's work).
[0039] Figure 1 The screen time of multiple devices is shown graphically. The screen time of each device is plotted on a timeline of a period (e.g., 24 hours). The screen time is shown as a line for each screen interaction instance, and the length of each line corresponds to the screen time of the device. For example, an instance 110 of the screen time of a desktop computer is shown in the first figure 101. The screen time of instance 110 can be calculated as the time elapsed between the start 111 (i.e., start) of instance 110 and the end 112 (i.e., stop) of instance 110. The cumulative screen time of all devices (i.e., across devices) is also plotted as the total screen time within period 100. The figure is presented to help understand. It should be understood that the devices and screen times shown are hypothetical and represent a possible implementation of many possible changes. For example, the screen time of additional and / or different devices may be included.
[0040] As shown, a first graph 101 shows an example of screen time corresponding to a first device (i.e., a desktop computer), a second graph 102 shows an example of screen time corresponding to a second device (i.e., a television), a third graph 103 shows an example of screen time associated with a third device (i.e., a mobile phone), and a fourth graph 104 shows an example of screen time associated with a fourth device (i.e., a tablet computer). A fifth graph 105 shows an example of screen time for all devices (i.e., across all devices). The instances of screen time can be totaled to calculate the total screen time within a period.
[0041] Total screen time can be calculated in a variety of ways. In one possible implementation, total screen time can be calculated as a cumulative total of accumulated screen time. In another possible implementation, total screen time can be the sum of all instances of screen time measured during a period. Thus, screen time can be measured as time (e.g., minutes) or rate (e.g., minutes / period).
[0042] In a possible implementation, measuring screen time may include a screen timer. For example, a screen timer may be started and stopped for each instance of screen time during a period. The screen timer may then be reset at the end of the period and restarted at the beginning of a subsequent period. The period 100 may be configured by a user (e.g., a parent) or set by a factory based on requirements (e.g., health guidelines). The period 100 may also be adjustable. For example, the period may end when the total screen time is reached. For example, screen time may accumulate until a threshold is reached, at which point the period ends.
[0043] Total screen time tracking can be customized to calculate a cumulative total of screen time based on device, time period (e.g., time of day), and / or location. For example, screen time for a user's work device can be tracked at times corresponding to workdays, and in possible implementations, total screen time for the work device can be calculated for each day, week, or month. In addition, a time period corresponding to a particular screen time tracking can be triggered based on the location of the head mounted device (i.e., the user). For example, a time period corresponding to a workday can begin when the user arrives at a location corresponding to work. Users based on device, time, and / or location.
[0044] Figure 2 2 is a perspective view of a head-mounted device according to a possible implementation of the present disclosure. As shown in the figure, the head-mounted device 200 can be implemented as smart glasses (e.g., AR glasses) configured to be worn on the user's head. The head-mounted device 200 includes a left lens and a right lens coupled to the user's ears through a left temple and a right temple, respectively. The user (i.e., the wearer) can observe the world through the left lens and the right lens, and the left lens and the right lens are coupled together by a nose bridge configured to rest on the wearer's nose.
[0045] The head mounted device 200 may include a sensing device configured to help determine where the user's focus is directed. For example, the head mounted device 200 may include at least one front camera. The front camera 230 may include an image sensor that can detect the intensity level of visible light and / or near infrared light (i.e., NIR light). The front camera 230 may be directed toward a front field of view (i.e., front FOV 235), or may include an optical device to route light from the front FOV 235 to the image sensor. For example, the front camera 230 may be positioned on the front surface of the head mounted device and include at least one lens to create an image of the front field of view (front FOV 235) on the image sensor. The front FOV 235 may include all (or part) of the user's field of view, so that an image or video of the world from the user's point of view (POV) may be captured by the front camera 230.
[0046] The head mounted device 200 may further include at least one eye tracking camera. The eye tracking camera 220 may be directed toward an eye field of view (i.e., eye FOV 225), or may include optics to route light from the eye FOV 225 to an eye image sensor. For example, the eye tracking camera 220 may be directed toward the user's eye and include at least one lens to create an image of the eye FOV 225 on the eye image sensor. The eye FOV 225 may include all (or part) of the field of view of the user's eye, such that an image or video of the eye is captured. The image of the eye may be analyzed by a processor (not shown) of the head mounted device to determine where the user is looking. For example, the relative position of the pupil in the image of the eye may correspond to the user's gaze direction.
[0047] The head mounted device may further include at least one orientation sensor 250. The orientation sensor may be implemented as a combination of an accelerometer, a gyroscope, and a magnetometer, which are combined to form an inertial measurement unit (i.e., an IMU) to determine the orientation of the head mounted device. The IMU may be configured to provide multiple measurements describing the orientation and movement of the head mounted display. For example, the IMU may have 6 degrees of freedom (6-DOF), which may describe three translational movements along the axes of the world coordinate system 260 (i.e., x direction, y direction, or z direction), and may describe three rotational movements around the axes of the world coordinate system 260 (i.e., pitch, yaw, roll). Sensor fusion may be used to combine data from the IMU with information about the earth's magnetic field to determine the orientation of the head mounted device coordinate system 270 relative to the world coordinate system 260. Information from the front camera 230, the eye FOV 225, and the IMU 250 may be combined to determine where the user's focus is directed, which may enable augmented reality applications. The head mounted display may further include interface devices for these applications as well.
[0048] The head-mounted device 200 may further include a human-machine interface subsystem configured to present information to the user. For example, the head-mounted device 200 may include a display configured to display information (e.g., text, graphics, images) in a display area 240 in one or both lenses. The display area may be all or a portion of the lens and may be visually transparent or translucent so that the user can view through the display area when not in use. The head-mounted device 200 may further include one or more speakers (e.g., earplugs 255) configured to play sounds (e.g., voice, music, tones). The present disclosure describes systems and methods for configuring sensing devices and interface devices of a head-mounted device for screen time measurement and response (e.g., alarms).
[0049] Figure 3 is a block diagram of a head mounted device according to a possible implementation of the present disclosure. As previously described, the head mounted device 200 includes a front camera 230, which is directed toward the user's viewpoint and is configured to capture an image of a front FOV 235. The head mounted device 200 further includes an eye tracking camera 220, which is directed toward one eye (or both eyes) of the user and is configured to capture an image of an eye FOV 225. The head mounted device 200 further includes at least one orientation sensor 250 (e.g., an IMU) configured to sense movement, position, and / or orientation of a head of a person wearing the head mounted device 200.
[0050] The head mounted device 200 may further include at least one position sensor 310. The at least one position sensor may be configured to determine the position of the head mounted device (i.e., the user). The position sensor 310 may include an ultra-wideband (UWB) sensor. The position sensor 310 may communicate with the screen device 317 via a communication link 315. For example, the head mounted device 200 and the screen device 317 may exchange information packets via a UWB communication link to determine the relative positions of the devices. For example, the position sensor may be configured to determine the round trip time (RTT) of the packet to be transmitted between the devices. The distance between the head mounted device and the screen device 317 may be sensed based on the round trip time. In addition, the position sensor may include a plurality of receivers configured to receive packets transmitted from the screen device 317. The position sensor may be configured to determine the time at which the packet arrives at the receiver to determine the angle between the screen device 317 and the position sensor 310. The position sensor may further include a global positioning system (GPS) sensor, which may be used to determine the geographic location of the head mounted device (i.e., the user). The geographic location may be further determined through sensor fusion methods, where information from a local area network (e.g., a WiFi network) and / or a cellular network may further refine the geographic location.
[0051] The head mounted device 200 further includes at least one processor 330. The processor communicates with the camera, sensors and other modules and electronics of the head mounted device. The processor is configured by instructions (e.g., software, applications, etc.) to perform a multi-modal method of tracking screen time across devices. The instructions may be non-transitory computer readable instructions stored in and called from the memory 340. Alternatively, the instructions may be transmitted to the processor from a computer coupled to the network 370 via the communication interface 350.
[0052] The processor 330 may communicate with the battery 360. The processor 330 may be configured by instructions to monitor the battery level and determine the mode of the head mounted device based on the battery level. For example, when the battery level is below a threshold, it may be determined that the head mounted device is in a low power mode, and when the battery level is above a threshold, it may be determined that the head mounted device is in a high power mode.
[0053] The processor may communicate with the display 320 of the head mounted device 200. The processor 330 may be configured by instructions to transmit text, graphics, video, images, etc. to the display 320. For example, the processor 330 may be configured to present an alert to the user on the display 320 in response to the measured screen time.
[0054] Figure 4 4 is a flow chart of a multi-mode screen time tracking method according to a possible implementation of the present disclosure. Method 400 includes detecting 410 a mode (i.e., screen time mode) that a head-mounted device will use to measure screen time. Detecting the screen time mode (i.e., mode) can be implemented in a variety of ways. In one possible implementation, detecting 410 the mode may include receiving input from a user specifying a mode (i.e., user input 402). In another possible implementation, detecting 410 the mode may include receiving the mode from a device setting 403 of the head-mounted device, where the device setting (i.e., system setting) may be configurable by an application (e.g., an AR application) running on the head-mounted device. In another possible implementation, detecting 410 the mode may include determining the mode based on a battery level 401 of the head-mounted device to balance the tracking quality and the operating time of the head-mounted device.
[0055] In a possible implementation, the detected mode may be a high power mode 490, a low power mode 600, or a no camera mode 900. Each mode may have different accuracy and power consumption. For example, the head mounted device 200 may consume more power in high power mode 490 than in low power mode 600. High power mode 490 may generate screen time with higher accuracy (i.e., confidence) than low power mode 600. No camera mode 900 may be used without any imaging, which may provide privacy for the user and provide the lowest power consumption mode. However, no camera mode 900 may be optional because it may rely on position sensors and orientation sensors that may or may not be present in all screen devices.
[0056] In a possible implementation, screen time tracking can alternate between modes depending on the need for accuracy or battery life. For example, screen time tracking for a device can start in high power mode 490 and then move to low power mode 600 between instances or within the same instance. This can help identify the device or screen content at the beginning of an instance of screen time with a higher confidence before moving to low power mode, which can track screen time more efficiently. In another example, when a screen is detected in low power mode, a high power mode can be triggered. This can help improve accuracy only when it has been determined with a certain confidence that the screen is being viewed.
[0057] like Figure 4 As shown in the implementation of , high power mode 490 uses the front camera 230 of the head mounted device and the image recognition algorithm 424 to determine the start (i.e., start) and end (i.e., stop) of an instance of screen time. Low power mode 600 uses the eye tracking camera 220 of the head mounted device and a neural network 434 (e.g., a U-net neural network) to determine the start / end of an instance of screen time. No camera mode 900 uses the position / orientation sensor 443 of the head mounted device and a machine learning model (e.g., sensor fusion) to determine the start / end of an instance of screen time.
[0058] Each mode can trigger the screen timer 430 to turn on / off according to the start / stop of the instance. In other words, the screen timer 430 can be used to determine the screen time regardless of the mode of starting / stopping the instance used to determine the screen time. For example, when it is determined that the screen is in the user's field of view, the screen timer can be activated (i.e., started), and then when it is determined that the screen is no longer in the user's field of view, the screen timer can be deactivated (i.e., stopped). Depending on the mode, different approaches can be taken to perform the determination of screen viewing conditions. The screen timer 430 can be reset at the end of the instance. Alternatively, the screen timer 430 can be turned on / off without resetting to accumulate screen time for multiple instances within a predetermined period of time, and then reset at the end of the predetermined period of time.
[0059] like Figure 4 As shown, method 400 may include a high power mode 490. High power mode 490 uses the front camera 230 of the head mounted device, which consumes more power than the eye tracking camera 220. In low power mode 600, the front camera 230 may be configured to be turned on (i.e., enabled, powered on, etc.) to capture one or more images corresponding to the user's field of view. For example, the front camera 230 captures a video (e.g., a high resolution color video) of items within the user's field of view. The image (or images) may be analyzed by an image recognition algorithm 424 to identify (i.e., detect) the screen (or screens) that the user is viewing. Detection of the screen may trigger the screen timer 430 to start timing.
[0060] The high power mode 490 process described above can be repeated periodically (i.e., looped) to track the screen in subsequently captured images to determine the screen time of the instance. Determining the screen time of the instance can include determining when the screen timer 430 should be stopped (i.e., deactivated, turned off). For example, when it has been determined that the screen has not been detected in the image for a plurality of cycles, the screen timer can be turned off. Each cycle can be separated by a cycle period. For example, the high power mode 490 can be repeated with a cycle period every few seconds (e.g., 2 seconds). When the instance has ended, the screen timer can output the screen time of the instance for recording and / or monitoring.
[0061] like Figure 4As shown, method 400 may include low power mode 600. Low power mode 600 uses the eye tracking camera 220 of the head mounted device, which consumes less power than the front camera. In low power mode 600, eye tracking camera 220 can be configured to be turned on (i.e., enabled, powered on, etc.) to capture one or more images corresponding to one eye (or multiple eyes) of the user. For example, eye tracking camera 220 can capture a video of the user's eyes. The image of the eye can be analyzed by a neural network to detect reflections and identify the screen in the reflection. The identification of the screen can trigger screen timer 430 to start timing.
[0062] Figure 5 yes Figure 4 Flowchart block diagram of a possible neural network for a low power mode of a multi-modal screen time tracking method shown in . The neural network is a U-net neural network 500, which is configured to receive an eye image 510 (i.e., an input image) from an eye tracking camera 220. The U-net neural network 500 can be configured to detect reflections of the image of the eye and identify the screen in the reflection. The identification can include generating a mask including the reflection, and applying the mask to the eye image 510 of the eye to generate a screen image 520 (i.e., an output image). In other words, the U-net neural network can be configured to predict whether the screen appears in the eye image, and if so, predict the position of the screen in the eye image (i.e., the center position of the screen).
[0063] Figure 6 6 shows a screen image identified in a reflection in an eye image according to a possible implementation of the present disclosure. As shown, a reflection can be detected in an eye image 610. A screen can be identified in the reflection by a neural network to generate a screen image 620. In this possible implementation, the screen is the entire reflection. In other words, the screen can be identified based on a U-net neural network 500 (see FIG. 5 ) configured to locate the screen in the image. Figure 5 ), a screen image 620 is generated by masking (i.e., segmenting) the eye image 610.
[0064] return Figure 4, the low power mode 600 process described above can be repeated periodically (i.e., looped) to track the screen in subsequently captured images to determine the screen time of the screen viewing instance. Determining the screen time of the instance can include determining when the screen timer 430 should be stopped (i.e., deactivated, turned off). For example, when it has been determined that the screen is not detected in the image within a plurality of cycles, the screen timer can be turned off. Each cycle can be separated by a cycle period. For example, the low power mode can be repeated with a cycle period every few seconds (e.g., 5 seconds). The cycle period of the low power mode can be longer than the cycle period of the high power mode to save power. The cycle period can be selected as a balance between screen time accuracy and power consumption. In a possible implementation, the low power mode 600 can be the default mode of the method.
[0065] like Figure 4 As shown, the method 400 may include a camera-free mode 900. The camera-free mode 900 uses the position / orientation sensor 443 of the head-mounted device to determine the orientation data of the user and the relative position data of the device. The orientation data of the user and the relative position data of the device (i.e., the screen) may be analyzed by the machine learning model 444 to determine that a screen viewing condition (i.e., a viewing condition) exists. For example, a viewing condition may exist when the orientation of the head-mounted device points in the direction of the device having the screen. Detection of the viewing condition may trigger the screen timer 430 to start timing. As previously described, the relative position data may include distances and angles calculated based on the use of ultra-wideband (UWB) communication. For example, the distance between UWB devices may be calculated based on the round-trip time of the UWB packets transmitted between the UWB devices. In addition, the angle between the UWB devices may be calculated based on the time difference of the UWB packets arriving at the UWB receivers spaced apart on the UWB devices receiving the UWB packets.
[0066] The above-described no-camera mode 900 process may be repeated (i.e., looped) periodically to track viewing conditions to determine the screen time of a screen viewing instance. Determining the screen time of an instance may include determining when the screen timer 430 should be stopped (i.e., deactivated, turned off). For example, when it has been determined that viewing conditions are not detected in the position / orientation data for a plurality of loops, the screen timer 430 may be turned off. Each loop may be separated by a loop period. For example, the low power mode may be repeated with a loop period of every few seconds (e.g., 5 seconds).
[0067] like Figure 4As shown, after the screen time has been calculated, it can be output for use in the recording / monitoring 450 process of method 400. The recording / monitoring process 450 can include recording the screen time determined from the screen timer into a database 750. The recording / monitoring process 450 can further include linking 455 the screen time with the device (i.e., classifying the screen time as being for a device). The recording / monitoring process 450 can further include generating 465 an alert based on one or more screen times stored in the database 750. For example, the screen time of an instance can be output from the screen timer in high power mode, low power mode, or no camera mode for classification, recording, and evaluation to generate an alert.
[0068] Figure 7 A flowchart of a method for recording screen time according to a possible implementation of the present disclosure is shown. The method 700 includes receiving screen time (e.g., screen time of an instance) acquired using a head mounted device in a high power mode, a low power mode, or a camera-less mode. The method 700 further includes classifying 710 the screen time into screen time corresponding to the device.
[0069] Classification 710 can utilize machine learning model 715. Machine learning model 715 can receive device information 720 acquired during an instance to link screen time 701 to a class of devices (e.g., laptop, desktop, TV, etc.) or a specific device (e.g., John's phone, living room TV, etc.). Device information can affect the confidence of the classification. For example, if a hand is identified in the image holding a screen, the model can reduce the confidence that the screen is a TV and increase the confidence that the screen is a mobile phone or tablet.
[0070] The device information 720 may include screen characteristics 721 derived from the captured image (e.g., FOV image). In a possible implementation, the screen characteristics 721 include physical or electronic properties of the screen derived from the image, such as screen size (e.g., length, width), screen resolution, or screen refresh rate. In another possible implementation, the screen characteristics 721 may include properties of the surrounding environment of the screen derived from the image, such as the housing of the screen or the hand holding the screen.
[0071] The device information 720 may further include screen content 722 derived from the captured image (e.g., eye image). In possible implementations, the screen content may include motion, color, image, etc., which hint at the type of screen being viewed. For example, a specific user interface may be identified and used to determine the device.
[0072] Device information 720 may further include a device identification (device ID 723) derived from the location data. In a possible implementation, device ID 723 may include information (e.g., address) identifying information communicated via UWB packets. For example, the device may be determined using an address communicated in a packet transmitted from the device.
[0073] Device information 720 may further include user location 724. The user location may be a location derived from a location service (e.g., GPS, indoor positioning, etc.). User location 724 may be a room. Knowledge of the room may inform decisions to be made by the machine learning model. For example, a TV is more likely to be located in the living room than in the bathroom. In other words, the method may include receiving a location of a head mounted device; and selecting a possible device based on the location.
[0074] The method for recording screen time may further include verifying 730 whether the classification made by the machine learning model 715 is correct or incorrect. Verification can be performed in a variety of ways. In one possible implementation, an image from a front camera can be captured (e.g., during a low power mode). The FOV image can be analyzed to determine whether the classification of the device is correct. In another possible implementation, the verification may include asking the user to classify the device (e.g., when no results from the model are returned) or confirming that the classification is correct. The results of the verification can be used to update the model 740. For example, updating the 740 model may include adjusting the model so that the confidence level of the classification generated from certain device information 720 changes based on the instance of the verification or based on the verification rate (e.g., false recognition rate, true recognition rate).
[0075] The method for recording screen time may further include recording the (classified) screen time in a database 750. Recording may include updating (i.e., increasing) the screen time of the device by the screen time 701 received (or derived) from the screen timer. Thus, recording 745 may include querying the database to obtain an entry corresponding to the device, and then adding the screen time of the device to the entry or creating a new entry for the device.
[0076] Back to Figure 4 , the multi-modal screen time tracking method 400 may further include generating an alert based on the screen time recorded in the database 750, and transmitting the alert to the display of the head mounted device. Generating an alert may include querying the database based on one or more rules corresponding to one or more alerts. For example, a rule may specify that the screen time of a particular device should not exceed a certain value within a period of time. Generating an alert may include querying the database 750 to determine the screen time of the particular device, and then comparing the screen time to a threshold. If the comparison shows that the screen time of the particular device reaches or exceeds the threshold, an alert may be generated. Various alerts may be generated.
[0077] The alert may include a message presented to the user regarding screen time. For example, the message may instruct the user to "pause" use of the device. The alert may include a report of screen time usage over a period (or multiple periods). The alert may include a comparison of first screen time for a first period of time and second screen time for a second period of time. These types of alerts may be presented to a display of the head mounted device for viewing by the user.
[0078] Figure 8 is an example of an alarm based on screen time. The alarm may include daily screen time for all devices (e.g., desktops, phones, tablets, etc.), which is shown in a bar graph for each day of the week. The alarm may further include the accumulated screen time for each device in the device within a week. The alarm may further include the total screen time for all devices within a week. The alarm may further include an analysis of the screen time, such as the average screen time per day. The alarm may further include the change in screen time compared to the previous period (i.e., last week). The alarm may be triggered to display at the end of the period (e.g., a week), or may be triggered by the user. In some implementations, an alarm may be generated when one or more of the screen times in the screen time meet one or more criteria.
[0079] In the specification and / or drawings, typical embodiments have been disclosed. The present disclosure is not limited to such exemplary embodiments. The use of the term "and / or" includes any and all combinations of one or more associated listed items. The drawings are schematic illustrations and are therefore not necessarily drawn to scale. Unless otherwise indicated, specific terms are used in a general and descriptive sense and not for limiting purposes.
[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as those generally understood by those of ordinary skill in the art. Methods and materials similar or equivalent to those described herein can be used to practice or test the present disclosure. As used in this specification and in the appended claims, the singular forms "one", "a", "the" include plural referents unless the context clearly indicates otherwise. The term "comprising" and its variations as used herein are used synonymously with the term "including" and its variations and are open, non-restrictive terms. The term "optional" or "optionally" used herein means that the features, events or situations described subsequently may or may not occur, and means that the description includes instances in which the features, events or situations occur and instances in which the features, events or situations do not occur. The range can be expressed in this article as from "about" a specific value and / or to "about" another specific value. When expressing such a range, on the one hand, it includes from a specific value and / or to another specific value. Similarly, when a value is expressed as an approximation, by using the antecedent "about", it should be understood that the specific value forms another aspect. It will be further understood that the endpoints of each of the ranges are significant both in relation to the other endpoint, and independently of the other endpoint.
[0081] Some implementations may be implemented using various semiconductor processing and / or packaging technologies. Some implementations may be implemented using various types of semiconductor processing technologies associated with semiconductor substrates, including but not limited to, for example, silicon (Si), gallium arsenide (GaAs), gallium nitride (GaN), silicon carbide (SiC), and the like.
[0082] Although certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the implementations. It should be understood that they are presented only by way of example and not limitation, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.
[0083] It should be understood that in the foregoing description, when an element is referred to as being on another element, connected to another element, electrically connected to another element, coupled to another element, or electrically coupled to another element, the element may be directly on another element, connected or coupled to another element, or one or more intermediate elements may be present. On the contrary, when an element is referred to as being directly on another element, directly connected to another element, or directly coupled to another element, there is no intermediate element. Although the term "directly on...", "directly connected to..." or "directly coupled to..." may not be used throughout the detailed description, the element that is shown as being directly on, directly connected to, or directly coupled to may be referred to as such. The claims of the present application (if any) may be modified to narrate the exemplary relationships described in the specification or shown in the drawings.
[0084] As used in this specification, singular forms may include plural forms unless the context clearly indicates a particular case. Spatially relative terms (e.g., on, above, upper, below, below, under, lower, etc.) are intended to include different orientations of the device in use or operation in addition to the orientation depicted in the figure. In some implementations, the relative terms above ... and below ... may include vertically above and vertically below, respectively. In some implementations, the term "adjacent" may include laterally adjacent or horizontally adjacent.
Claims
1. A head mounted device, comprising: an eye-tracking camera directed toward the user's eyes; as well as a processor in communication with the eye tracking camera, the processor being configured via instructions to: capturing an eye image of the eye of the user using the eye tracking camera; analyzing the eye image to identify screen reflections; After identifying the screen reflection, starting a screen timer; capturing subsequent eye images at regular intervals to track the screen reflection over time; When the screen reflection can no longer be tracked, stopping the screen timer; as well as A screen time of the first instance is recorded based on the screen timer.
2. The head mounted device according to claim 1, wherein: Analyzing the eye image to identify screen reflections is performed by a neural network.
3. A head mounted device according to any one of the preceding claims, wherein: The processor is further configured to: classifying the screen time of the first instance as being for a first device based on a machine learning model configured with device information captured by the head mounted device; and The screen time for the first instance of the first device is recorded in a database.
4. The head mounted device according to claim 3, wherein: The device information is any combination of a device identifier, content of the screen reflection, characteristics of the screen reflection, and a location of the user.
5. A head mounted device according to any one of the preceding claims, wherein: The processor is further configured by instructions to: Detecting that a mode of the head mounted device for screen time tracking is a low power mode.
6. The head mounted device according to claim 5, further comprising: The front camera, and wherein: The processor is further configured to: Detecting that the mode of the head mounted device for screen time tracking is a high power mode: Capturing a visual field image of the user's visual field using the front camera; analyzing the visual field image to identify a screen; After identifying the screen in the visual field image, starting the screen timer; capturing subsequent field of view images at regular intervals to track the screen over time; When the screen can no longer be tracked, stopping the screen timer; and A screen time of a second instance is recorded based on the screen timer.
7. The head mounted device according to claim 6, further comprising: at least one orientation sensor configured to sense orientation data of the head mounted device; as well as at least one position sensor configured to sense relative position data of the device, and wherein: The processor is further configured to: detecting that the mode of the head mounted device for screen time tracking is in a camera-less mode for screen time tracking; and in the camera-less mode, the processor is configured to: capturing the orientation data of the head mounted device and the relative position data of the device; analyzing the orientation data and the relative position data to identify a viewing condition; After identifying the viewing condition, starting the screen timer; periodically capturing subsequent orientation data and subsequent relative position data to track the viewing condition over time; When the viewing condition can no longer be tracked, stopping the screen timer; and A screen time of a third instance is recorded based on the screen timer.
8. The head mounted device according to claim 7, wherein: The at least one orientation sensor comprises an inertial measurement unit (IMU); and The at least one position sensor comprises an ultra-wideband (UWB) sensor.
9. The head mounted device according to claim 7 or claim 8, wherein: The processor is further configured to: generating an alert based on the screen time of the first instance, the screen time of the second instance, and / or the screen time of the third instance; and The alert is displayed on a display of the head mounted device.
10. A computer-implemented method comprising: Detecting that the mode of the headset used for screen time tracking is low power mode; capturing an eye image of an eye using an eye tracking camera of the head mounted device; analyzing the eye image to identify screen reflections; After identifying the screen reflection, starting a screen timer; capturing subsequent eye images at regular intervals to track the screen reflection over time; as well as When the screen reflection can no longer be tracked, the screen timer is stopped.
11. The computer-implemented method of claim 10, wherein: Periodically capturing subsequent eye images to track the screen reflection over time includes: The capturing and the analyzing are repeated with a cycle period of greater than 2 seconds.
12. A computer-implemented method according to claim 10 or claim 11, wherein: Detecting that the mode of the head mounted device for screen time tracking is a low power mode includes: detecting a battery level of the head mounted device; and It is determined that the battery level is below a threshold.
13. A computer-implemented method according to any one of claims 10 to 12, wherein: Analyzing the eye image to identify screen reflections includes: applying the eye image to an input of a neural network, the eye image comprising the eye and the screen reflection; and A screen image is received at an output of the neural network, the screen image including the screen reflection.
14. The computer-implemented method according to any one of claims 10 to 13, further comprising: detecting that the mode of the head mounted device for screen time tracking is a high power mode; Capturing a field of view image using a front-facing camera of the head-mounted device; analyzing the visual field image to identify a screen; After identifying the screen, starting the screen timer; capturing subsequent field of view images at regular intervals to track the screen over time; as well as When the screen can no longer be tracked, the screen timer is stopped.
15. The computer-implemented method of claim 14, wherein: Detecting that the mode of the head mounted device for screen time tracking is a high power mode includes: detecting a battery level of the head mounted device; and It is determined that the battery level is above a threshold.
16. The computer-implemented method of claim 14 or claim 15, further comprising: detecting that the mode of the head mounted device for screen time tracking is a no-camera mode; capturing orientation data of the head mounted device using at least one orientation sensor of the head mounted device; capturing relative position data of the device from ultra-wideband (UWB) signals received at the head mounted device; analyzing the orientation data and the relative position data to determine viewing conditions; After identifying the viewing condition, starting a screen timer; periodically capturing subsequent orientation data and relative position data to track the viewing conditions over time; as well as When the viewing condition can no longer be tracked, the screen timer is stopped.
17. The computer-implemented method of claim 16, further comprising: recording screen time in a database based on the screen timer; as well as An alert is generated when the screen time meets criteria.
18. The computer-implemented method of claim 17, further comprising: Based on device information captured by the head mounted device, instances of screen time are classified as corresponding to a device.
19. A head mounted device comprising: an eye-tracking camera directed toward the user's eyes; A front camera, the front camera pointing to the user's field of view; at least one orientation sensor configured to sense orientation data of the head mounted device worn by the user; as well as A processor, the processor being configured by instructions to: Detecting a low power mode based on a battery level of the head mounted device, wherein in the low power mode: The screen time of the device is determined based on eye images from the eye tracking camera applied to a neural network; and detecting a high power mode based on the battery level of the head mounted device, wherein in the high power mode: The screen time of the device is determined based on images from the front-facing camera applied to an image recognition algorithm.
20. The head mounted device according to claim 19, wherein: The processor is further configured by instructions to: Detecting a no-camera mode based on a user input, wherein in the no-camera mode: The screen time of the device is based on the orientation data of the head-mounted device and relative position data of the device, which is captured from an ultra-wideband (UWB) signal received at the head-mounted device and applied to a machine learning model.
21. A head mounted device according to claim 19 or claim 20, wherein: The head-mounted device is augmented reality glasses.
