Interaction-based user suggestions

Abstract

Systems and processes for operating a digital assistant are provided. An example method includes: receiving, at an electronic device having one or more processors and memory, a biometric signature of a user; determining, based on the biometric signature, a health level associated with the user; in accordance with a determination that the health level associated with the user is below a predetermined threshold, providing, to the user, a first output for ceasing to interact with the first electronic device, wherein the first electronic device is capable of producing a virtual environment; and after a predetermined time, causing provision, to the user, of a second output for beginning to interact with the first electronic device, utilizing a second electronic device.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 174,199, filed April 13, 2021, entitled “USER SUGGESTIONS BASED ON ENGAGEMENT,” the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] This invention relates generally to digital assistants, and more specifically to monitoring user interactions and providing suggestions based on user interactions with digital assistants. Background Technology

[0004] Intelligent automated assistants (or digital assistants) provide a beneficial interface between human users and electronic devices. Such assistants allow users to interact with devices or systems using natural language in voice and / or text. For example, a user can provide voice input containing their request to a digital assistant running on an electronic device. The digital assistant can interpret the user's intent from this voice input and act it out as a task. These tasks can then be performed by executing one or more services of the electronic device, and relevant output in response to the user's request can be returned to the user.

[0005] In some situations, users may interact with digital assistants in such a comprehensive way within virtual and augmented reality settings that they may neglect monitoring their own well-being and therefore be unable to determine when they should take a break from interacting with the digital assistant and / or electronic devices. Therefore, an effective way for digital assistants to monitor users' well-being and prompt them to take a break is desirable. It may also be expected that digital assistants provide prompts for users to end their breaks and resume interaction to improve user productivity and engagement levels. Summary of the Invention

[0006] This document discloses an exemplary method. An exemplary method includes: receiving a user's biometric signature at an electronic device having one or more processors and memory; determining a health level associated with the user based on the biometric signature; providing the user with a first output to stop interacting with a first electronic device, wherein the first electronic device is capable of generating a virtual environment, based on the determination that the health level associated with the user is below a predetermined threshold; and, after a predetermined time, causing a second output to be provided to the user using a second electronic device to start interacting with the first electronic device.

[0007] This document discloses an example non-transitory computer-readable medium. An exemplary non-transitory computer-readable storage medium stores one or more programs. The one or more programs include instructions for: receiving a user's biometric signature; determining a health level associated with the user based on the biometric signature; providing the user with a first output to stop interacting with a first electronic device, wherein the first electronic device is capable of generating a virtual environment, based on the determination that the health level associated with the user is below a predetermined threshold; and, after a predetermined time, causing a second output to be provided to the user using a second electronic device to start interacting with the first electronic device.

[0008] This document discloses an example electronic device. An example electronic device includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a user's biometric signature; determining a health level associated with the user based on the biometric signature; providing the user with a first output to stop interacting with the first electronic device if the determined health level associated with the user is below a predetermined threshold, wherein the first electronic device is capable of generating a virtual environment; and, after a predetermined time, causing a second output to be provided to the user using a second electronic device to begin interacting with the first electronic device.

[0009] An exemplary electronic device includes: means for receiving a user's biometric signature; means for determining a health level associated with the user based on the biometric signature; means for providing the user with a first output to stop interacting with the first electronic device if the health level associated with the user is determined to be below a predetermined threshold, wherein the first electronic device is capable of generating a virtual environment; and means for providing the user with a second output to start interacting with the first electronic device using a second electronic device after a predetermined time.

[0010] Determining a user's associated health level based on biometric signatures allows digital assistants to monitor whether the user is effectively interacting with them and any selected tasks the assistant is performing. Therefore, the digital assistant can determine whether the user will benefit from breaks in interaction and thus use their time more effectively. This, in turn, improves the efficiency of the digital assistant and any electronic device running it, as users waste less time repeating tasks they don't interact with. This allows the digital assistant to provide tasks when the user is engaged, rather than repeatedly performing tasks when the user is distracted, thereby reducing the digital assistant's power consumption and improving the battery life of electronic devices. Attached Figure Description

[0011] Figures 1A-1B Exemplary systems for various computer-generated reality technologies, including virtual reality and mixed reality, are depicted.

[0012] Figure 2 Exemplary systems for monitoring a user's health level and providing user suggestions are described, based on various examples.

[0013] Figure 3 An exemplary output of a system used to monitor a user's health level is depicted.

[0014] Figure 4 An exemplary output of a system used to monitor a user's health level is depicted.

[0015] Figure 5 An exemplary output of a system used to monitor a user's health level is depicted.

[0016] Figure 6 An exemplary flowchart is depicted, illustrating methods for monitoring a user's health level and providing user suggestions based on various examples. Detailed Implementation

[0017] People can sense or interact with the physical environment or world without using electronic devices. Physical features such as physical objects or surfaces can be included within the physical environment. For example, a physical environment may correspond to a physical city with physical buildings, roads, and vehicles. People can directly perceive or interact with the physical environment through various means such as smell, sight, taste, hearing, and touch. This is in contrast to extended reality (XR) environments, which can refer to partially or fully simulated environments that people can sense or interact with using electronic devices. XR environments can include virtual reality (VR) content, mixed reality (MR) content, augmented reality (AR) content, and so on. Using an XR system, physical movement of a person or a portion of their representation can be tracked, and in response, the properties of virtual objects in the XR environment can be altered in a manner that conforms to at least one law of nature. For example, an XR system can detect head movement of a user and adjust the auditory and graphical content presented to the user in a way that simulates how sound and view would change in a physical environment. In other examples, an XR system can detect movement of electronic devices (e.g., laptops, tablets, mobile phones, etc.) that present the XR environment. Therefore, XR systems can simulate how sound and view will change in the physical environment to adjust the auditory and graphical content presented to the user. In some instances, other inputs, such as representations of body movement (e.g., voice commands), can enable XR systems to adjust the properties of the graphical content.

[0018] Numerous types of electronic systems allow users to sense or interact with their XR environment. An incomplete list of examples includes lenses with integrated display capabilities placed on the user's eyes (e.g., contact lenses), head-up displays (HUDs), projection-based systems, head-mounted systems, windows or windshields with integrated display technology, headsets / earpieces, input systems with or without haptic feedback (e.g., handheld or wearable controllers), smartphones, tablets, desktop / laptop computers, and speaker arrays. Head-mounted systems may include opaque displays and one or more speakers. Other head-mounted systems may be configured to receive from opaque external displays, such as the opaque external display of a smartphone. Head-mounted systems may use one or more image sensors to capture images / video of the physical environment, or one or more microphones to capture audio of the physical environment. Some head-mounted systems may include transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may guide light representing an image to the user's eyes through media such as holographic media, optical waveguides, optical combiners, optical reflectors, other similar technologies, or combinations thereof. Various display technologies can be used, such as liquid crystal on silicon, LED, uLED, OLED, laser scanning light sources, digital light projection, or combinations thereof. In some examples, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection technology, which projects images onto a user's retina, or can project virtual content into a physical environment, such as onto a physical surface or as a hologram.

[0019] Figure 1A and Figure 1B An exemplary system 100 for various extended reality technologies is depicted.

[0020] like Figure 1A As shown, system 100 includes device 100a. Device 100a includes RF circuitry 104, processor 102, memory 106, image sensor 108, touch-sensitive surface 122, speaker 118, position sensor 116, microphone 112, orientation sensor 110, and display 120. These components optionally communicate using communication bus 150 of device 100a.

[0021] In some examples, a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop) implements some components of system 100, and a second device (e.g., a head-mounted device) implements other components of system 100. In some examples, device 100a is implemented in either the base station device or the second device.

[0022] like Figure 1BAs shown, in some examples, system 100 includes two or more devices that communicate, for example, via a wired or wireless connection. A first device 100b (e.g., a base station device) includes a memory 106, RF circuitry 104, and a processor 102. Such components optionally communicate using a communication bus 150 of device 100b. A second device 100c (e.g., a head-mounted device) includes components such as RF circuitry 104, processor 102, memory 106, image sensor 108, touch-sensitive surface 122, speaker 118, position sensor 116, microphone 112, orientation sensor 110, and display 120. These components optionally communicate using a communication bus 150 of device 100c.

[0023] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with networks (e.g., the Internet, wireless networks (e.g., such as cellular networks and wireless local area networks (LANs)) and / or intranets) and / or electronic devices. RF circuitry 104 optionally includes circuitry for using near-field communication and / or short-range communication (e.g., ...). The circuit used for communication.

[0024] System 100 includes a processor 102 and a memory 106. The processor 102 includes one or more graphics processors, one or more general-purpose processors, and / or one or more digital signal processors. In some examples, the memory 106 is one or more non-transitory computer-readable storage media (e.g., random access memory, flash memory) configured to be executed by the processor 102 to perform computer-readable instructions described below.

[0025] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the physical environment. For example, an active IR sensor includes an IR emitter (e.g., an IR point emitter) for emitting infrared light into the physical environment. Image sensor 108 also optionally includes one or more visible light image sensors, such as complementary metal-oxide-semiconductor (CMOS) sensors and / or charge-coupled device (CCD) sensors, capable of acquiring images of physical elements from the physical environment. Image sensor 108 also optionally includes one or more event cameras configured to capture movement of physical elements in the physical environment. Image sensor 108 also optionally includes one or more depth sensors capable of detecting distances to physical elements of system 100. In some examples, system 100 uses the IR sensor, CCD sensor, event camera, and depth sensor together to detect the physical environment surrounding system 100. In some examples, image sensor 108 includes a first image sensor and a second image sensor. The first and second image sensors are optionally capable of capturing images of physical elements in the physical environment from two corresponding different perspectives. In some examples, system 100 uses image sensor 108 to detect the position and orientation of system 100 and / or display 120 in a physical environment. For example, system 100 uses image sensor 108 to track the position and orientation of display 120 relative to one or more fixed elements in the physical environment. In some examples, image sensor 108 is capable of receiving user input, such as gestures.

[0026] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user input such as tap and swipe input. In some examples, the touch-sensitive surface 122 and display 120 are combined to form a touch-sensitive display.

[0027] In some examples, system 100 includes a microphone 112. System 100 uses microphone 112 to detect physical environment from a user or sound from a user. In some examples, microphone 112 includes a microphone array (e.g., including multiple microphones) that optionally operates together, for example, to locate spatial sound sources from the physical environment or identify ambient noise.

[0028] System 100 includes an orientation sensor 110 for detecting the orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in the position and / or orientation of system 100 and / or display 120 (such as relative to physical elements in the physical environment). Orientation sensor 110 optionally includes a gyroscope and / or accelerometer.

[0029] System 100 includes a display 120. Display 120 may operate in conjunction with a transparent or translucent display (and optionally with one or more imaging sensors). Display 120 may include an opaque display. Display 120 may allow a person to directly observe the physical environment through the display, and may also allow virtual content to be added to the person's field of view, for example, by overlaying virtual content onto the physical environment. Display 120 may implement display technologies such as digital light projectors, laser scanning light sources, LEDs, OLEDs, liquid crystal on silicon, or combinations thereof. Display 120 may include a substrate through which light transmits, such as optical reflectors and combiners, optical waveguides, holographic substrates, or combinations thereof. As a particular example, the transparent or translucent display may selectively transition between a transparent or translucent state and an opaque state. Other exemplary embodiments of display 120 include a display-capable lens, tablet computer, smartphone, desktop computer, laptop computer, head-up display, display-capable automotive windshield, or display-capable window. In some examples, system 100 is a projection-based system. For example, system 100 projects virtual objects onto a physical environment (e.g., projects a hologram onto a physical environment or projects an image onto a physical surface). As another example, system 100 uses retinal projection to project an image onto a person's eye (e.g., the retina). In some examples, system 100 may be configured to interface with an external display (e.g., a smartphone display).

[0030] Figure 2 Exemplary systems 200 for monitoring user health levels and providing user suggestions are depicted, based on various examples. In some examples, such as Figure 2 As shown, system 200 includes a digital assistant 202, a VR device 204, a wearable electronic device 206, and an output device 208. In some examples, these components of system 200 may be optionally combined, as discussed further below. In some examples, digital assistant 202 is implemented on one or more devices included in system 200, such as VR device 204, wearable electronic device 206, and / or output device 208. In some examples, digital assistant 202 is implemented across devices other than those depicted in system 200 (e.g., servers). In some examples, some modules and functions of the digital assistant are divided into server and client portions, wherein the client portion resides on one or more user devices (e.g., electronic device 100, VR device 204, wearable electronic device 206, and output device 208) and communicates with the server portion via one or more networks.

[0031] It should be noted that system 200 is merely one example of such a system, and system 200 may have more or fewer devices than shown, may combine two or more devices, or may have different device configurations or arrangements. Digital assistant 202 is implemented in hardware, software instructions for execution by one or more processors, firmware (including one or more signal processing integrated circuits and / or application-specific integrated circuits), or combinations thereof. In some examples, digital assistant 202 connects to one or more components and / or sensors of VR device 204, wearable electronics 206, and output device 208, as discussed further below.

[0032] System 200 uses various depicted devices (including VR device 204, wearable electronic device 206, and output device 208) to monitor the health level of the user of System 200 to determine whether the user should take a break, and thus determine whether System 200 should provide the user with output to stop interacting with System 200. Additionally, System 200 may use various devices to provide the user with other outputs or reminders, including prompts to resume interaction with System 200 (or a specific device of System 200) when specific criteria are met (e.g., after a break, when the user has scheduled an event, or when the user is able to work efficiently).

[0033] When a user interacts with VR device 204, system 200, and specifically VR device 204 of system 200, receives the user's biometric signature 201. Biometric signature 201 includes one or more biometric signals that may indicate the user's health level, including the user's pulse, blood pressure, attention level, eye pattern, facial pattern (e.g., facial expression), voice pattern, electroencephalogram (EEG), or body temperature. Biometric signature 201 is received by one or more sensors of VR device 204, including one or more cameras, a thermometer, a heart rate sensor, one or more microphones, one or more electrodes, etc.

[0034] For example, when a user wears VR device 204 (e.g., VR headset), a camera on VR device 204, pointed towards the user's face, can monitor the user's eyes to determine the user's eye map. As another example, when a user wears VR device 204, the earpiece or strap used to hold VR device 204 may include a heart rate sensor and / or thermometer that can measure the user's pulse or body temperature, respectively.

[0035] In some examples, biometric signature 201 includes multiple biometric signatures received by multiple different sensors of VR device 204. For example, biometric signature 201 may include the user's eye pattern determined by the internal camera of VR device 204, the user's body temperature determined by the thermometer of VR device 204, the user's voice pattern received by the microphone of VR device 204, and the user's pulse received by the heart rate sensor of VR device 204.

[0036] In some examples, the biometric signature 201 is received by one or more sensors of the wearable device 206 or output device 208 and provided to the VR device 204 or another electronic device (e.g., a server) for further processing by the digital assistant 202. For example, when the wearable device 206 is a smartwatch, the wearable device 206 may use a heart rate monitor included in the smartwatch to measure the user's pulse and provide the user's pulse as the biometric signature 201 to the VR device 204.

[0037] Upon receiving the biometric signature 201, the VR device 204 provides the biometric signature 201 to the digital assistant 202, which determines a health level 203 associated with the user based on the biometric signature 201. The digital assistant 202 may be implemented on the VR device 204 and thus uses various components of the VR device 204 (e.g., memory, processor, etc.) to determine the health level 203. In some examples, the digital assistant 202 is implemented on both a server and the VR device 204, and thus uses components of both the VR device 204 and the server (e.g., memory, processor, etc.) to determine the health level 203.

[0038] In some examples, health level 203 includes a score determined based on biometric signature 201. Therefore, digital assistant 202 can assign a value to each biometric signature received as part of biometric signature 201 and average them to determine the user's overall health level. For example, when biometric signature 201 includes the user's pulse and eye pattern, digital assistant 202 can assign a value to the user's pulse based on whether the pulse is considered normal, high, etc., and also assign a value to the user's eye pattern based on whether the eye pattern is unstable. Therefore, if the user's pulse exceeds 100 (e.g., 110, 115, 125, 145, etc.), a lower value can be assigned to the pulse, indicating that the user may be in poor health. Similarly, if the user's eye pattern is very unstable (e.g., looking at many different objects and directions), a lower value can be assigned to the eye pattern, also indicating that the user may be in poor health. Therefore, averaging these two values ​​will result in a low value for health level 203, indicating that the user has a low overall health level.

[0039] In contrast, if a user's pulse is below 100 (e.g., 80, 85, 90, etc.), a higher value can be assigned to the pulse, indicating that the user may be in good health. Similarly, if a user's eye diagram is not very unstable (e.g., focused on a single object, document, etc.), a higher value can be assigned to the eye diagram, indicating that the user may be in good health. Therefore, averaging these two values ​​will result in a high health level of 203, indicating that the user has a high overall health level.

[0040] In some examples, health level 203 is based on one or more interactions between the user and VR device 204. These interactions include opening a new task or window, shifting attention away from the primary task, adjusting settings on VR device 204, etc. Interactions such as those described above may indicate that the user is losing focus and may have been interacting with VR device 204 for a longer period than normally expected, resulting in a lower health level. Therefore, when the user has already interacted with VR device 204 in this manner, digital assistant 202 may determine a relatively low health level 203 (or adjust health level 203 to an even lower level).

[0041] In some examples, health level 203 is based on contextual data associated with the user. Exemplary contextual data associated with the user includes the amount of time the user has interacted with VR device 204, one or more scheduled events associated with the user, user profiles associated with the user, or other information associated with the user that digital assistant 202 can access. Therefore, when determining health level 203, digital assistant 202 may take this contextual data into account to calculate or adjust the score.

[0042] For example, contextual data associated with the user could indicate that the user has been interacting with the VR device 204 for several hours, and that the user has scheduled a virtual meeting one hour from now. The digital assistant 202 could then determine or adjust the health level 203 to a lower level to indicate that the user should take a break, even if the biometric signal 201 indicates that the user is generally in good health.

[0043] In some examples, health level 203 is based on contextual data associated with VR device 204. The contextual data associated with VR device 204 may indicate the location of VR device 204 (e.g., GPS coordinates), whether VR device 204 is connected to a network (e.g., a WiFi network), whether VR device 204 is connected to one or more other devices (e.g., headphones), and / or the current time, date, and / or weekday. If VR device 204 is connected to a network or device, the contextual data may also indicate the name and / or type of the network or device. Therefore, when determining health level 203, digital assistant 202 may consider this contextual data to calculate or adjust the score.

[0044] For example, contextual data associated with VR device 204 could indicate that VR device 204's current location is a public park and that VR device 204 is not connected to a WiFi network. Therefore, digital assistant 202 could determine that the user is in a good position to take a break and thus determine (or adjust) health level 203 to a lower level to indicate that the user should take a break, even if biometric signal 201 indicates that the user is generally well.

[0045] Contextual data associated with VR device 204 may also include the number of open applications, the types of open applications, or specific objects / data provided by the open applications. Therefore, digital assistant 202 may also determine or adjust health level 203 based on this application data. For example, contextual data associated with VR device 204 may indicate that the user has ten different applications open and all applications are sharing different, unrelated objects or data. Based on this, digital assistant 202 may determine (or adjust) health level 203 to a lower level to indicate that the user should take a break, because the contextual data indicates that the user may be overwhelmed, even if this is not reflected in biometric signal 201.

[0046] The context data associated with VR device 204 may also include the presence or active connection of one or more other devices of system 200 (e.g., wearable device 206 and output device 208) to VR device 204 at its current location. For example, the context data associated with VR device 204 may indicate the presence of one or more output devices 208 (e.g., smart speakers) at VR device 204's current location. Therefore, digital assistant 202 can determine that now is a good time for the user to take a break, as it can remind the user to re-engage with output device 208 later, as further described below. Thus, digital assistant 202 can adjust health level 203 to a lower level to encourage the user to take a break from interacting with VR device 204.

[0047] Health level 203 is also based on whether VR device 204 is currently providing a virtual environment. For example, if VR device 204 is currently providing a virtual environment, digital assistant 202 can give greater weight to factors such as how long the user has been interacting with VR device 204. Specifically, the current provision of a virtual environment combined with long usage time can indicate that the user has not recently taken a break, and therefore digital assistant 202 can adjust health level 203 accordingly.

[0048] In some examples, health level 203 is based on contextual data associated with other devices of system 200, such as wearable device 206 and output device 208. The contextual data associated with wearable device 206 and output device 208 is similar to the contextual data associated with VR device 204 and can indicate the location of the electronic device (e.g., GPS coordinates), whether the electronic device is connected to a network (e.g., a WiFi network), whether the electronic device is connected to one or more other devices (e.g., headphones), and the presence of one or more other devices of system 204. Therefore, as discussed above, digital assistant 202 can recognize that because the user is near other devices of system 200, the user is more likely to respond to an alert from one of those devices, and that now is a good time for the user to rest.

[0049] After determining health level 203, digital assistant 202 determines whether health level 203 is below a predetermined threshold. The predetermined threshold is a threshold that indicates whether the user's current condition is good or bad. For example, the threshold can be any number that can indicate a boundary between good and bad conditions based on health level 203. Therefore, the predetermined threshold can be any real number, such as 50, 60, 75, 95, 100, etc.

[0050] It should be understood that while this document describes the determination that health level 203 is below a predetermined threshold, system 200 and digital assistant 202 can also determine whether health level 203 is above or below the predetermined threshold, and perform further actions as described below based on that determination. Therefore, system 200 is not limited to simply determining that health level 203 is below a predetermined threshold, and may include flexible determinations depending on when the user wants to be notified or whether someone other than the user (such as a medical professional) wants to be notified based on a specific health level. Furthermore, system 200 may include customized notifications for specific health levels or specific biometric signatures that the user or medical professional may find helpful or interesting.

[0051] In some examples, the predetermined thresholds are determined based on one or more default thresholds. For example, default values ​​may be provided by a doctor or other expert panel, indicating when the user is generally healthy or in good condition and therefore does not need to take a break from interacting with the VR device 204. Thus, these values ​​provided (determined) by experts can be applied to system 200 as predetermined thresholds to indicate when the user is in good or bad condition.

[0052] In some examples, the predetermined threshold is determined based on one or more baseline biometric signatures associated with the user. In some examples, one or more baseline biometric signatures associated with the user are determined based on previous interactions between the user and VR device 204. For example, when the user interacts with VR device 204, digital assistant 202 may record biometric signature 201. This can occur across many different instances of interaction between the user and VR device 204, allowing digital assistant 202 to determine the user's biometric signature when the user is at a good rest and therefore at an appropriate baseline.

[0053] In some examples, one or more baseline biometric signatures associated with the user are determined based on the setup process of VR device 204 and / or digital assistant 202. For example, when a user first uses VR device 204, digital assistant 202 may prompt the user to complete the setup process, where digital assistant 202 requests the user to provide various biometric signatures, such as pulse, body temperature, voice signature, etc. These biometric signatures can then be stored as baseline biometric signatures and used to determine predetermined thresholds or modified over time, as discussed above.

[0054] In some examples, one or more baseline biometric signatures associated with a user are stored in a user profile associated with the user that can be accessed by the digital assistant 202. For example, as discussed above, when a user provides a baseline biometric signature while setting up the VR device 204, the baseline biometric signature can be stored in the user profile associated with the user. The user profile can then be updated over time with new biometric signatures to provide a more accurate baseline biometric signature. In some examples, the user profile may exist before setting up the VR device 204, and baseline biometric data can be added to a previously existing user profile. In some examples, the user profile can be created when the baseline biometric data is determined (e.g., during the setup of the VR device 204).

[0055] When the digital assistant 202 determines that the health level 203 is below a predetermined threshold, the digital assistant 202 provides the user with an output (e.g., a first output) to stop interacting with the VR device 204. In some examples, the output to stop interacting with the VR device 204 includes visual cues provided within the virtual environment, such as...Figure 3 As shown. Therefore, the digital assistant 202 can provide a prompt 302, including "Now is a good time to take a break!", within the virtual environment 300, which includes various virtual objects (including virtual objects 304 and 306). In this way, the user's interaction with the VR device 204 is interrupted by this prompt.

[0056] In some examples, VR device 204 may detect input to de-prompt 302, such as a virtual swipe of prompt 302 or audio input saying "I'll take a break later." In response to detecting input to de-prompt 302, digital assistant 202 may de-prompt 302 and restore the virtual environment 300, allowing the user to continue interacting with VR device 204. In some examples, after de-prompt 302, digital assistant 202 may abandon providing prompts for a predetermined time (e.g., 10, 15, 30, or 45 minutes) and then provide prompts again. In some examples, after de-prompt 302, digital assistant 202 may abandon providing prompts until the health level 203 is again determined to be below a predetermined threshold.

[0057] In some examples, the output for stopping interaction with VR device 204 includes haptic feedback. For example, in addition to or instead of the cue provided 302 discussed above, VR device 204 may also provide the user with an output for stopping interaction with VR device 204 via vibration. In some examples, haptic feedback is provided using a second device of system 200, such as wearable device 204. For example, when wearable device 204 is a smartwatch, digital assistant 202 may enable wearable device 204 to provide haptic feedback, instead of cue provided by VR device 204 or in addition to cue provided by VR device 204.

[0058] In some examples, the output used to stop interaction with the VR device 204 (e.g., the first output) is determined based on one or more user settings. For example, a user might indicate that they prefer an audio output rather than... Figure 3 Visual indicators are used to provide cues. Therefore, when the digital assistant 202 determines that an output should be provided to the user, it will provide that output as audio. In some examples, one or more user settings are stored in a user profile associated with the user. For example, information indicating the user's preferred audio output may be stored in the user profile associated with the user, along with other data such as the baseline biometric signature discussed above. In some examples, the output used to stop interaction with the VR device 204 is of the default type, such as providing visual cues in the virtual environment.

[0059] After providing an output to stop interaction with VR device 204, digital assistant 202 waits for a predetermined time and then causes another electronic device (e.g., wearable device 206 or output device 208) utilizing system 200 to provide a second output to the user. Therefore, the second output is a prompt for the user to interact with VR device 204 and the virtual environment provided by VR device 204 again.

[0060] In some examples, the scheduled time is based on one or more user settings. For instance, a user might specify the duration of their desired break, such as 10 minutes, 20 minutes, 30 minutes, 45 minutes, or 60 minutes. Therefore, the user can change the break length and thus the scheduled time, allowing a second output to be delivered when the user wishes. In some examples, the scheduled time is a default duration, such as 15 minutes.

[0061] In some examples, the scheduled time is based on contextual data associated with the user and / or device of system 200. As discussed above, the contextual data associated with the user includes the amount of time the user has interacted with VR device 204, one or more scheduled events associated with the user, user profiles associated with the user, or other user-related information accessible to digital assistant 202. Furthermore, the contextual data associated with the device of system 200 includes whether the device is connected to a network, the time of day, the date, the device's location, etc. Therefore, digital assistant 202 can determine or adjust the scheduled time based on one or more of the contextual data discussed above.

[0062] For example, contextual data associated with a user might instruct the user to attend a meeting at their office in one hour, while contextual data associated with VR device 204 might instruct VR device 204 to connect to the user's associated home network. Therefore, digital assistant 202 can determine that the user is at home and needs to arrive at their office within one hour. Thus, digital assistant 202 can set the scheduled time to the length of time required for the user to arrive at their office on time for the meeting, such as 30 minutes. Therefore, digital assistant 202 can provide a second output to the user after 30 minutes, allowing the user to re-interact with VR device 204 at this time and be reminded of the upcoming meeting.

[0063] In some examples, the second output includes haptic feedback. For instance, digital assistant 202 may provide haptic feedback at wearable device 206 to prompt the user to re-interact with VR device 204. In some examples, the second output includes audio output, such as... Figure 4 As shown. For example, digital assistant 202 can use one of the output devices 208, such as smart speaker 404, to provide the user with an audio prompt 402, "You look like you've rested well, let's start again!"

[0064] In some examples, the second output includes a visual cue on one of the output devices 208, such as... Figure 5 As shown. For example, digital assistant 202 may provide a visual cue 502 to the user on the screen 504 of device 500 (e.g., smartphone or tablet) saying, “You look refreshed, let’s get started!”

[0065] It should be understood that system 200 can provide any combination of the above-described outputs to prompt the user to stop interacting with VR device 204 and then to interact with VR device 204 again. Therefore, system 200 can provide both haptic feedback and visual cues, both visual cues and audio cues, both audio cues and haptic feedback, or any other combination of outputs to prompt the user to stop interacting with VR device 204 or to prompt the user to interact with VR device 204 again.

[0066] When the digital assistant 202 determines that the health level 203 is higher than a predetermined threshold, the digital assistant 202 ceases to provide any output and continues to provide a virtual environment using the VR device 204. In this way, the user can continue to use the VR device 204 and the digital assistant 202 as they wish until the user's health level falls below the predetermined threshold.

[0067] Figure 6 This is a flowchart illustrating methods for monitoring a user's health level and providing user recommendations, according to various examples. Method 600 is performed at a device (e.g., devices 100, 204, 206, 208) having one or more input devices (e.g., a touchscreen, microphone, camera) and wireless communication radios (e.g., Bluetooth connection, WiFi connection, mobile broadband connection such as 4G LTE connection). In some embodiments, the electronic device includes multiple cameras. In some examples, the device includes one or more biometric sensors, which optionally include cameras, such as infrared cameras, thermal imaging cameras, or combinations thereof. Some operations in method 600 may be optionally combined, some operations may be optionally changed in order, and some operations may be optionally omitted.

[0068] At step 602, a biometric signature (e.g., biometric signature 201) is received from the user. In some examples, the biometric signature includes the user's pulse, blood pressure, attention level, eye pattern, facial pattern (e.g., facial expression), voice pattern, EEG, or body temperature.

[0069] At step 604, a health level (e.g., health level 203) associated with the user is determined based on the biometric signature (e.g., biometric signature 201). In some examples, the health level includes a score based on the biometric signature received from one or more sensors of a first electronic device (e.g., electronic device 100, VR device 204, wearable device 206, output device 208). In some examples, the score is based on one or more interactions between the user and the first electronic device.

[0070] In some examples, the health level associated with a user (e.g., health level 203) is based on contextual data associated with the user. In some examples, the contextual data associated with the user includes the amount of time the user has interacted with a first electronic device (e.g., electronic device 100, VR device 204, wearable device 206, output device 208). In some examples, the contextual data associated with the user includes one or more scheduled events associated with the user.

[0071] In some examples, the health level associated with a user (e.g., health level 203) is based on contextual data associated with an electronic device (e.g., electronic device 100, VR device 204, wearable device 206, output device 208). In some examples, the contextual data associated with the first electronic device includes one or more applications open on the first electronic device. In some examples, the contextual data associated with the first electronic device includes the location of the first electronic device. In some examples, the health level associated with a user is based on whether the first electronic device is providing a virtual environment (e.g., virtual environment 300).

[0072] At step 606, based on the determination that the user's associated health level is below a predetermined threshold, a first output (e.g., prompts 302, 402, 502) is provided to the user to stop interacting with the first electronic device (e.g., electronic device 100, VR device 204, wearable device 206, output device 208). In some examples, the first electronic device is capable of generating a virtual environment (e.g., virtual environment 300). In some examples, the predetermined threshold is determined based on one or more baseline biometric signatures associated with the user (e.g., biometric signature 201). In some examples, one or more baseline biometric signatures associated with the user are determined based on previous interactions between the user and the first electronic device.

[0073] In some examples, the predetermined threshold is determined based on a default threshold. In some examples, the first output (e.g., cue 302, 402, 502) includes a visual cue within a virtual environment (e.g., virtual environment 300). In some examples, the first output includes haptic feedback. In some examples, the first output is determined based on one or more user settings.

[0074] At step 608, after a predetermined time, a second electronic device (e.g., electronic device 100, VR device 204, wearable device 206, output device 208) provides the user with a second output (e.g., prompts 302, 402, 502) to initiate interaction with the first electronic device (e.g., electronic device 100, VR device 204, wearable device 206, output device 208). In some examples, the predetermined time is based on one or more user settings. In some examples, the predetermined time is based on one or more default settings. In some examples, the second electronic device is a wearable electronic device. In some examples, both the first and second electronic devices are associated with the user's user profile. In some examples, the second output is audio output.

[0075] As described above, one aspect of the present invention involves collecting and using data from various sources to improve the delivery of content that users may be interested in. This disclosure contemplates that, in some instances, such collected data may include personal information that uniquely identifies or can be used to contact or locate specific individuals. Such personal information may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0076] This disclosure recognizes that the use of such personal information data in the techniques of this invention can benefit users. For example, the personal information data can be used to deliver targeted content that is of interest to the user. Therefore, the use of such personal information data enables users to have planned control over the delivered content. Furthermore, this disclosure also anticipates other uses of personal information data that are beneficial to users. For example, health and fitness data can be used to provide insights into a user's overall health status or as positive feedback for individuals using technology to pursue health goals.

[0077] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as data collection and / or use change. Personal information from users should be collected for the entity's lawful and reasonable purposes and not shared or sold outside of these lawful uses. Furthermore, such collection / sharing should be conducted only after obtaining informed consent from users. Additionally, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and processes. Furthermore, such entities may be subject to third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Additionally, policies and practices should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including specific considerations regarding jurisdiction. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while in other countries, health data may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0078] Regardless of the foregoing, this disclosure also envisions examples of users selectively blocking the use or access to personal information data. That is, this disclosure contemplates providing hardware and / or software components to prevent or block access to such personal information data. For example, with regard to information delivery services, the technology of the present invention can be configured to allow users to opt-in or opt-out at any time during or after service registration to participate in the collection of personal information data. In another example, users may choose not to provide user information to the delivery service. In yet another example, users may choose to limit the length of time their user information is retained. In addition to providing opt-in and opt-out options, this disclosure also envisions providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0079] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0080] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed examples, it also contemplates that various examples can be implemented without access to such personal information data. That is, various examples of the inventive technology will not be rendered inoperable due to the lack of all or part of such personal information data. For example, preferences can be inferred based on non-personal information data or a minimal amount of personal information such as content requested by a device associated with a user, other non-personal information available to the content delivery service, or publicly available information, thereby selecting content and delivering it to the user.

Claims

1. A method for providing user-based suggestions based on interaction, comprising: At a first electronic device having memory and one or more processors: The user's biometric signature is received while the user is interacting with the first electronic device via one or more sensors associated with the first electronic device; The health level associated with the user is determined based on the biometric signature and contextual data associated with the user; Based on the determination that the health level associated with the user is below a predetermined threshold, a first output is provided to the user to stop interacting with the first electronic device, wherein the first electronic device is capable of generating a virtual environment; as well as After a predetermined time, a second output is provided to the user using a second electronic device, wherein the second output relates to a renewed interaction with the first electronic device.

2. The method of claim 1, wherein the biometric signature includes the user's attention level.

3. The method according to any one of claims 1 to 2, wherein the health level includes a score based on the biometric signature received from the one or more sensors of the first electronic device.

4. The method of claim 3, wherein the score is based on one or more interactions between the user and the first electronic device.

5. The method of claim 1, wherein the context data associated with the user includes the amount of time the user has interacted with the first electronic device.

6. The method of claim 1, wherein the context data associated with the user includes one or more scheduled events associated with the user.

7. The method of claim 1, wherein the health level associated with the user is based on contextual data associated with the electronic device.

8. The method of claim 7, wherein the context data associated with the first electronic device includes one or more applications opened on the first electronic device.

9. The method of claim 7, wherein the context data associated with the first electronic device includes the location of the first electronic device.

10. The method of claim 1, wherein the health level associated with the user is based on whether the first electronic device is providing a virtual environment.

11. The method of claim 1, wherein the predetermined threshold is determined based on one or more baseline biometric signatures associated with the user.

12. The method of claim 11, wherein the one or more baseline biometric signatures associated with the user are determined based on previous interactions between the user and the first electronic device.

13. The method of claim 1, wherein the predetermined threshold is determined based on a default threshold.

14. The method of claim 1, wherein the first output includes visual cues within the virtual environment.

15. The method of claim 1, wherein the first output includes haptic feedback.

16. The method of claim 1, wherein the first output is determined based on one or more user settings.

17. The method of claim 1, wherein the predetermined time is based on one or more user settings.

18. The method of claim 1, wherein the predetermined time is based on one or more default settings.

19. The method of claim 1, wherein the second electronic device is a wearable electronic device.

20. The method of claim 1, wherein both the first electronic device and the second electronic device are associated with the user's user profile.

21. The method of claim 1, wherein the second output is an audio prompt.

22. An electronic device, comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: The user's biometric signature is received while the user is interacting with the first electronic device via one or more sensors associated with the first electronic device; The health level associated with the user is determined based on the biometric signature and contextual data associated with the user. Based on the determination that the health level associated with the user is below a predetermined threshold, a first output is provided to the user to stop interacting with the first electronic device, wherein the first electronic device is capable of generating a virtual environment; as well as After a predetermined time, a second output is provided to the user using a second electronic device, wherein the second output relates to a renewed interaction with the first electronic device.

23. The electronic device of claim 22, wherein the biometric signature includes the user's level of attention.

24. The electronic device according to any one of claims 22 to 23, wherein the health level includes a score based on the biometric signature received from the one or more sensors of the first electronic device.

25. The electronic device of claim 24, wherein the score is based on one or more interactions between the user and the first electronic device.

26. The electronic device of claim 22, wherein the context data associated with the user includes the amount of time the user has interacted with the first electronic device.

27. The electronic device of claim 22, wherein the context data associated with the user includes one or more scheduled events associated with the user.

28. The electronic device of claim 22, wherein the health level associated with the user is based on contextual data associated with the electronic device.

29. The electronic device of claim 28, wherein the context data associated with the first electronic device includes one or more applications opened on the first electronic device.

30. The electronic device of claim 28, wherein the context data associated with the first electronic device includes the location of the first electronic device.

31. The electronic device of claim 22, wherein the health level associated with the user is based on whether the first electronic device is providing a virtual environment.

32. The electronic device of claim 22, wherein the predetermined threshold is determined based on one or more baseline biometric signatures associated with the user.

33. The electronic device of claim 32, wherein the one or more baseline biometric signatures associated with the user are determined based on previous interactions between the user and the first electronic device.

34. The electronic device of claim 22, wherein the predetermined threshold is determined based on a default threshold.

35. The electronic device of claim 22, wherein the first output includes visual cues within the virtual environment.

36. The electronic device of claim 22, wherein the first output includes haptic feedback.

37. The electronic device of claim 22, wherein the first output is determined based on one or more user settings.

38. The electronic device of claim 22, wherein the predetermined time is based on one or more user settings.

39. The electronic device of claim 22, wherein the predetermined time is based on one or more default settings.

40. The electronic device of claim 22, wherein the second electronic device is a wearable electronic device.

41. The electronic device of claim 22, wherein both the first electronic device and the second electronic device are associated with the user's user profile.

42. The electronic device of claim 22, wherein the second output is an audio prompt.

43. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: The user's biometric signature is received while the user is interacting with the first electronic device via one or more sensors associated with the first electronic device; The health level associated with the user is determined based on the biometric signature and contextual data associated with the user; Based on the determination that the health level associated with the user is below a predetermined threshold, a first output is provided to the user to stop interacting with the first electronic device, wherein the first electronic device is capable of generating a virtual environment; as well as After a predetermined time, a second output is provided to the user using a second electronic device, wherein the second output relates to a renewed interaction with the first electronic device.

44. The computer-readable storage medium of claim 43, wherein the biometric signature includes the user's level of attention.

45. The computer-readable storage medium according to any one of claims 43 to 44, wherein the health level comprises a score based on the biometric signature received from the one or more sensors of the first electronic device.

46. ​​The computer-readable storage medium of claim 45, wherein the score is based on one or more interactions between the user and the first electronic device.

47. The computer-readable storage medium of claim 43, wherein the context data associated with the user includes the amount of time the user has interacted with the first electronic device.

48. The computer-readable storage medium of claim 43, wherein the context data associated with the user includes one or more scheduled events associated with the user.

49. The computer-readable storage medium of claim 43, wherein the health level associated with the user is based on context data associated with the electronic device.

50. The computer-readable storage medium of claim 49, wherein the context data associated with the first electronic device includes one or more applications opened on the first electronic device.

51. The computer-readable storage medium of claim 49, wherein the context data associated with the first electronic device includes the location of the first electronic device.

52. The computer-readable storage medium of claim 43, wherein the health level associated with the user is based on whether the first electronic device is providing a virtual environment.

53. The computer-readable storage medium of claim 43, wherein the predetermined threshold is determined based on one or more baseline biometric signatures associated with the user.

54. The computer-readable storage medium of claim 53, wherein the one or more baseline biometric signatures associated with the user are determined based on previous interactions between the user and the first electronic device.

55. The computer-readable storage medium of claim 43, wherein the predetermined threshold is determined based on a default threshold.

56. The computer-readable storage medium of claim 43, wherein the first output includes visual cues within the virtual environment.

57. The computer-readable storage medium of claim 43, wherein the first output includes haptic feedback.

58. The computer-readable storage medium of claim 43, wherein the first output is determined based on one or more user settings.

59. The computer-readable storage medium of claim 43, wherein the predetermined time is based on one or more user settings.

60. The computer-readable storage medium of claim 43, wherein the predetermined time is based on one or more default settings.

61. The computer-readable storage medium of claim 43, wherein the second electronic device is a wearable electronic device.

62. The computer-readable storage medium of claim 43, wherein both the first electronic device and the second electronic device are associated with the user's user profile.

63. The computer-readable storage medium of claim 43, wherein the second output is an audio prompt.

64. An electronic device comprising: Apparatus for performing the method according to any one of claims 1 to 21.

Images