Receiving notifications while operating in different modes

Abstract

The present disclosure relates generally to systems and methods of presenting an augmented reality environment, and more particularly to displaying notifications while operating in different modes while presenting an augmented reality environment. In some cases, an electronic device detects an event that meets one or more first criteria, including criteria that are met when an interaction with a person is detected using at least an optical sensor. In some examples, in response to detecting the event, the electronic device transitions from a first mode to a second mode. In some examples, the electronic device receives a notification while operating in the second mode. In some examples, if one or more second criteria are met, the electronic device does not present the notification while in the second mode.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 585,184 filed on September 25, 2023, U.S. Provisional Application No. 63 / 669,639 filed on July 10, 2024, and U.S. Patent Application No. 18 / 893,788 filed on September 23, 2024, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present invention generally relates to systems and methods for receiving notifications at an electronic device when the electronic device is operating in different modes, and more particularly to receiving notifications when operating in different modes on an extended reality device. Background Art

[0004] Some computer graphics environments provide two-dimensional environments and / or three-dimensional environments in which at least some of the objects presented for user viewing are virtual and generated by a computer. In some examples, the computer graphics environment may be based on one or more images of the physical environment of the computer. A user may pre-program the mode of how a notification electronic device should present notifications when the electronic device is in a given mode. An electronic device is needed to automatically detect changes in the physical environment and automatically transition from a first operating mode to a second operating mode. Summary of the invention

[0005] The present invention generally relates to systems and methods for presenting an extended reality environment, and more specifically to displaying notifications while operating in different modes while presenting an extended reality environment. In some examples, presenting an extended reality environment with an electronic device includes presenting a pass-through video of the physical environment of the electronic device. As described herein, for example, presenting a pass-through video may include displaying a virtual or video pass-through, wherein the electronic device uses a display to display an image of the physical environment. In other examples, presenting an extended reality environment with an electronic device includes presenting a real or non-imagined optical perspective, wherein a portion of the physical environment is visible to the user through a transparent portion of the display. In some cases, the electronic device detects an event that satisfies one or more first criteria, the one or more first criteria including criteria that are satisfied when an interaction with a person is detected using at least an optical sensor. In some examples, in response to detecting an event, the electronic device transitions from a first mode to a second mode. In some examples, the electronic device receives a notification while operating in the second mode. In some examples, if one or more second criteria are met, the electronic device does not present a notification when in the second mode.

[0006] A comprehensive description of these examples is provided in the drawings and detailed description, and it should be understood that this summary does not limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 An electronic device presenting an extended reality environment according to some examples of the present disclosure is illustrated.

[0008] Figure 2 A block diagram illustrating an example architecture of a device according to some examples of the present disclosure.

[0009] FIG. 3A to FIG. 3D Examples of events satisfying one or more criteria and the resulting transition of an electronic device from a first mode to a second mode according to some examples of the present disclosure are illustrated.

[0010] FIG. 4A to FIG. 4C Examples of a second event satisfying one or more criteria and a resulting transition of the electronic device from a first mode to a third mode according to some examples of the present disclosure are illustrated.

[0011] Figure 5 Examples of notifications presented on a second electronic device but not on the electronic device according to some examples of the present disclosure are illustrated.

[0012] FIG. 6A to FIG. 6C An example of presenting a summary of notifications received when the electronic device is in the second mode according to some examples of the present disclosure is illustrated.

[0013] Figure 7 An example process is illustrated of how an electronic device according to some examples of the present disclosure detects an event and transitions a mode as a result of the event. DETAILED DESCRIPTION

[0014] The present invention generally relates to systems and methods for presenting an extended reality environment, and more specifically to displaying notifications while operating in different modes while presenting an extended reality environment. In some examples, presenting an extended reality environment with an electronic device includes presenting a see-through video of a physical environment of the electronic device. As described herein, for example, presenting a see-through video may include displaying a virtual or video see-through, wherein the electronic device uses a display to display an image of the physical environment. In other examples, presenting an extended reality environment with an electronic device includes presenting a real or non-imagined optical see-through, wherein a portion of the physical environment is visible to a user through a transparent portion of the display. In some cases, the electronic device detects an event that satisfies one or more first criteria, the one or more first criteria including a criterion that is satisfied when an interaction with a person is detected using at least an optical sensor. In some examples, in response to detecting an event, the electronic device transitions from a first mode to a second mode. In some examples, the electronic device receives a notification while operating in the second mode. In some examples, if one or more second criteria are met, the electronic device does not present a notification when in the second mode. In some examples, a three-dimensional object is displayed in a computer-generated three-dimensional environment in a specific orientation that controls one or more behaviors of the three-dimensional object (e.g., when the three-dimensional object moves within the three-dimensional environment). In some examples, the orientation of displaying the three-dimensional object in the three-dimensional environment is selected by a user of the electronic device or automatically selected by the electronic device. For example, when initiating presentation of a three-dimensional object in a three-dimensional environment, a user may select a particular orientation of the three-dimensional object, or the electronic device may automatically select an orientation of the three-dimensional object (eg, based on the type of three-dimensional object).

[0015] In some examples, a three-dimensional object may be displayed in a three-dimensional environment in a world-locked orientation, a body-locked orientation, a tilt-locked orientation, or a head-locked orientation, as described below. As used herein, an object displayed in a three-dimensional environment in a body-locked orientation has a distance and orientation offset relative to a portion of the user's body (e.g., the user's torso). Alternatively, in some examples, the body-locked object has a fixed distance from the user, and the orientation of the content does not reference any part of the user's body (e.g., may be displayed in the same basic direction relative to the user regardless of head and / or body movement). Additionally or alternatively, in some examples, the body-locked object may be configured to always maintain gravity or horizon (e.g., perpendicular to gravity) alignment, so that changes in the head and / or body in the rolling direction will not cause the body-locked object to move within the three-dimensional environment. Instead, translational movement in either configuration will cause the body-locked object to reposition within the three-dimensional environment to maintain the distance offset.

[0016] As used herein, an object displayed in a head-locked orientation in a three-dimensional environment has a distance and orientation offset relative to the user's head. In some examples, the head-locked object moves within the three-dimensional environment when the user's head moves (when the user's viewpoint changes).

[0017] As used herein, an object displayed in a world-locked orientation in a three-dimensional environment has no distance or orientation offset relative to the user.

[0018] As used herein, an object displayed in a three-dimensional environment with a tilt-locked orientation (referred to herein as a tilt-locked object) has a distance offset relative to a user, such as a portion of a user's body (e.g., a user's torso) or a user's head. In some examples, the tilt-locked object is displayed with a fixed orientation relative to the three-dimensional environment. In some examples, the tilt-locked object moves according to a polar (e.g., spherical) coordinate system centered on a pole passing through the user (e.g., the user's head). For example, the tilt-locked object is moved in the three-dimensional environment based on the movement of the user's head in a spherical space around the user's head (e.g., centered on the user's head). Thus, if the user tilts his head relative to gravity (e.g., up or down in a pitch direction), the tilt-locked object will follow the head tilt and move radially along the sphere, so that the tilt-locked object is repositioned within the three-dimensional environment to the same distance offset relative to the user as before the head tilt, while optionally maintaining the same orientation relative to the three-dimensional environment. In some examples, if the user moves his head in a rolling direction (e.g., clockwise or counterclockwise) relative to gravity, the tilt-locked object is not repositioned within the three-dimensional environment.

[0019] Figure 1 An electronic device 101 is illustrated that presents an extended reality (XR) environment (e.g., a computer-generated environment that optionally includes representations of physical and / or virtual objects) according to some examples of the present disclosure. In some examples, such as Figure 1 As shown, the electronic device 101 is a head mounted display or other head mounted device configured to be worn on the head of the user of the electronic device 101. Figure 2 An example of an electronic device 101 is described by using an architectural block diagram of FIG. Figure 1 As shown, electronic device 101 and table 106 are located in a physical environment. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lights, etc.). In some examples, electronic device 101 may be configured to use display 120 to detect and / or capture an image of the physical environment including table 106 (illustrated in the field of view of electronic device 101).

[0020] In some examples, such as Figure 1As shown in FIG. 1 , the display 120 includes one or more internal image sensors 114 a oriented toward the user's face (e.g., see below with reference to FIG. 1 ). Figure 2 100 ). In some examples, internal image sensor 114a is used for eye tracking (e.g., detecting the gaze of a user). Internal image sensor 114a is optionally arranged on the left and right portions of display 120 to enable eye tracking of the user's left and right eyes. In some examples, display 120 also includes external image sensors 114b and 114c facing outward from the user to detect and / or capture the physical environment of electronic device 101 and / or the movement of the user's hands or other body parts.

[0021] In some examples, display 120 has a field of view (e.g., a field of view captured by external image sensors 114b and 114c and / or visible to the user via display 120). Because display 120 is optionally part of a head mounted device, the field of view of display 120 is optionally the same as or similar to the field of view of the user's eyes. In some examples, display 120 is a transparent or semi-transparent display through which portions of the physical environment in the field of view of electronic device 101 pass. For example, the computer-generated environment includes an optical see-through or video pass-through portion of the physical environment in which electronic device 101 is located.

[0022] In some examples, in response to a trigger, the electronic device 101 may be configured to display in the XR environment Figure 1 , which does not exist in the physical environment but is displayed in XR on top of a real-world table 106 (or a representation thereof). Optionally, in response to detecting the plane of the table 106 in the physical environment 100, the virtual object 104 may be displayed on the surface of the table 106 in the XR environment displayed via the display 120 of the electronic device 101.

[0023] It should be understood that virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional XR environment. For example, a virtual object may represent an application or user interface displayed in an XR environment. In some examples, a virtual object may represent content corresponding to an application and / or displayed via a user interface in an XR environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures, such as air pinch gestures, air tap gestures, and / or air touch gestures), so that a user virtually touches, taps, moves, rotates, or otherwise interacts with virtual object 104.

[0024] In some examples, displaying an object in a three-dimensional environment may include interaction with one or more user interface objects in the three-dimensional environment. For example, initiation of display of an object in a three-dimensional environment may include interaction with one or more virtual options / indicators displayed in the three-dimensional environment. In some examples, when display of an object is initiated in a three-dimensional environment, an electronic device may track the user's gaze as an input for identifying one or more virtual options / indicators as targets for selection. For example, gaze may be used to identify one or more virtual options / indicators as targets for selection using another selection input. In some examples, a virtual option / indicator may be selected using hand tracking input detected via an input device that communicates with the electronic device. In some examples, an object displayed in a three-dimensional environment may be moved and / or reoriented in the three-dimensional environment based on movement input detected via an input device.

[0025] In the following discussion, an electronic device that communicates with a display generation component and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in the present disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device to be displayed on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as used in the present disclosure, input received on an electronic device (e.g., touch input received on a touch-sensitive surface of an electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, and the electronic device receives input information from the independent input device.

[0026] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and / or a digital video player application.

[0027] Figure 2A block diagram of an example architecture of an electronic device 201 according to some examples of the present disclosure is illustrated. In some examples, the electronic device 201 includes one or more electronic devices. For example, the electronic device 201 can be a portable device, an auxiliary device that communicates with another device, a head-mounted display, etc. In some examples, the electronic device 201 corresponds to the above reference Figure 1 An electronic device 101 is described.

[0028] like Figure 2 As illustrated in FIG. 1 , the electronic device 201 optionally includes various sensors, such as one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206 (optionally corresponding to Figure 1 114a and / or external image sensors 114b and 114c in the embodiment of the present invention), one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors), one or more display generation components 214 (optionally corresponding to Figure 1 The display 120 in the electronic device 201, one or more speakers 216, one or more processors 218, one or more memories 220 and / or communication circuits 222. One or more communication buses 208 are optionally used for communication between the above components of the electronic device 201.

[0029] The communication circuit 222 optionally includes circuits for communicating with electronic devices, networks such as the Internet, intranets, wired networks and / or wireless networks, cellular networks, and wireless local area networks (LANs). The communication circuit 222 optionally includes circuits for using near field communication (NFC) and / or short-range communication such as A circuit for communicating.

[0030] Processor 218 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 220 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processor 218 to perform the techniques, processes, and / or methods described below. In some examples, memory 220 may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in combination with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, a magnetic storage device, an optical storage device, and / or a semiconductor storage device. Examples of such storage devices include magnetic disks, optical disks based on compact disc (CD), digital versatile disc (DVD) or Blu-ray technology, and persistent solid-state memory (such as flash memory, solid-state drives, etc.).

[0031] In some examples, display generation component 214 includes a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation component 214 includes multiple displays. In some examples, display generation component 214 may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, electronic device 201 includes touch-sensitive surfaces 209 for receiving user inputs (such as tap inputs and swipe inputs or other gestures). In some examples, display generation component 214 and touch-sensitive surface 209 form a touch-sensitive display (e.g., a touch screen integrated with electronic device 201 or a touch screen that communicates with electronic device 201 outside electronic device 201).

[0032] The electronic device 201 optionally includes an image sensor 206. The image sensor 206 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real-world environment. The image sensor 206 also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensor 206 also optionally includes one or more cameras configured to capture the movement of physical objects in the real-world environment. The image sensor 206 also optionally includes one or more depth sensors configured to detect the distance of the physical object from the electronic device 201. In some examples, information from the one or more depth sensors may allow the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some examples, the one or more depth sensors may allow the device to determine the texture and / or topography of an object in the real-world environment.

[0033] In some examples, the electronic device 201 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around the electronic device 201. In some examples, the image sensor 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, the electronic device 201 uses the image sensor 206 to detect the positioning and orientation of the electronic device 201 and / or the display generation component 214 in the real-world environment. For example, the electronic device 201 uses the image sensor 206 to track the positioning and orientation of the display generation component 214 relative to one or more fixed objects in the real-world environment.

[0034] In some examples, the electronic device 201 includes a microphone 213 or other audio sensor. The electronic device 201 optionally uses the microphone 213 to detect sounds from the user and / or the user's real-world environment. In some examples, the microphone 213 includes a microphone array (multiple microphones) that optionally operate in conjunction to identify ambient noise or locate a sound source in the space of the real-world environment.

[0035] The electronic device 201 includes a location sensor 204 for detecting the location of the electronic device 201 and / or the display generating component 214. For example, the location sensor 204 may include a global positioning system (GPS) receiver that receives data from one or more satellites and allows the electronic device 201 to determine the absolute location of the device in the physical world.

[0036] The electronic device 201 includes an orientation sensor 210 for detecting the orientation and / or movement of the electronic device 201 and / or the display generation component 214. For example, the electronic device 201 uses the orientation sensor 210 to track changes in the position and / or orientation of the electronic device 201 and / or the display generation component 214, such as relative to physical objects in the real world environment. The orientation sensor 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0037] In some examples, the electronic device 201 includes a hand tracking sensor 202 and / or an eye tracking sensor 212 and / or other body tracking sensors, such as a leg tracking sensor, a torso tracking sensor, and / or a head tracking sensor. The hand tracking sensor 202 is configured to track the position / location of one or more parts of the user's hand, and / or the movement of one or more parts of the user's hand relative to the extended reality environment, relative to the display generation component 214, and / or relative to another defined coordinate system. The eye tracking sensor 212 is configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented together with the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented separately from the display generation component 214.

[0038] In some examples, the hand tracking sensor 202 (and / or other body tracking sensors, such as leg, torso, and / or head tracking sensors) may use an image sensor 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more body parts (e.g., the legs, torso, head, or hands of a human user). In some examples, the hand may be resolved with sufficient resolution to distinguish fingers and their corresponding positioning. In some examples, one or more image sensors 206 are positioned relative to the user to define a field of view and an interaction space of the image sensor 206, in which the finger / hand position, orientation, and / or movement captured by the image sensor is used as input (e.g., to distinguish from the user's resting hand or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0039] In some examples, eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source such as an LED) that emits light toward the user's eyes. The eye tracking camera may be pointed toward the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by corresponding eye tracking cameras and illumination sources, and focus / gaze may be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by one or more corresponding eye tracking cameras / illumination sources.

[0040] The electronic device 201 is not limited to Figure 2 Rather than having components and configurations, the electronic device 201 may include fewer components, other components, or additional components in a variety of configurations. In some examples, the electronic device 201 may be implemented between two electronic devices (e.g., as a system). In some such examples, each (or more) electronic device may each include one or more of the same components discussed above, such as various sensors, one or more display generation components, one or more speakers, one or more processors, one or more memories, and / or communication circuits. One or more people using the electronic device 201 are optionally referred to herein as one or more users of the device.

[0041] Attention is now directed to the interaction with one or more virtual objects displayed in a three-dimensional environment presented at an electronic device (e.g., corresponding to electronic device 201 and / or electronic device 101). In some examples, the electronic device adjusts the presentation of notifications in the three-dimensional environment according to detected events and mode transitions. As described below, the electronic device may use one or more input devices (e.g., image sensor 206, orientation sensor 210, audio sensor, and other sensors) to detect events, such as interactions with people at a specific location. In some examples, and as described below, the electronic device automatically changes the operating mode based on the detected event. In some examples, different modes control the way notifications are presented in a three-dimensional environment. It is time-consuming to manually switch the operating mode of the electronic device according to the event. Some existing electronic devices allow users to create and / or modify different operating modes that control the presentation of notifications in different ways. However, these existing devices do not automatically detect the current event and do not automatically switch the operating mode based on the detected event.

[0042] In order to solve the technical problems outlined above, an exemplary method and / or system is provided, in which an electronic device is configured to detect (e.g., and identify / classify) an event and thus transform the operation mode of the electronic device. When an event is detected for which a user does not want a notification to be presented, the electronic device automatically switches from a first mode (e.g., a current operation mode) to a second mode, so that the user does not need to manually mute the notification.

[0043] FIG. 3A to FIG. 3D Examples of events satisfying one or more first criteria and the resulting transition of an electronic device from a first mode to a second mode according to some examples of the present disclosure are illustrated. FIG. 3A to FIG. 3D To illustrate the process described below, including Figure 7 Process 700.

[0044] Figure 3A The electronic device 101 is illustrated presenting a three-dimensional environment 300 from the perspective of a user of the electronic device 101 (e.g., facing a table 302 in a room in which the electronic device 101 is positioned) via a display 120. In some examples, the user's viewpoint determines what content (e.g., physical and / or virtual objects) is visible in a viewport (e.g., a view of the three-dimensional environment 300 visible to the user via one or more displays 120, displays, or a pair of display modules that provide stereoscopic content to different eyes of the same user). In some examples, the (virtual) viewport has a viewport that defines the user's viewport via a display 120. FIG. 3A to FIG. 3DThe viewport boundaries of the range of the three-dimensional environment 300 visible on the display 120 in the display 120. In some examples, the area defined by the viewport boundaries is smaller than the user's visual range in one or more dimensions (e.g., based on the user's visual range, the size, optical properties or other physical characteristics of one or more displays, and / or the position and / or orientation of one or more displays relative to the user's eyes). In some examples, the area defined by the viewport boundaries is larger than the user's visual range in one or more dimensions (e.g., based on the user's visual range, the size, optical properties or other physical characteristics of one or more displays, and / or the position and / or orientation of one or more displays relative to the user's eyes). The viewport and viewport boundaries typically move as one or more displays move (e.g., for a head-mounted device, as the user's head moves, or for a handheld device such as a tablet computer or smart phone, as the user's hand moves). The user's viewpoint determines what is visible in the viewport, the viewpoint generally specifies a position and orientation relative to the three-dimensional environment, and as the viewpoint moves, the view of the three-dimensional environment will also move in the viewport. For head-mounted devices, the viewpoint is typically based on the position and orientation of the user's head, face, and / or eyes to provide a view of a three-dimensional environment that is perceptually accurate and provides an immersive experience when the user is using the head-mounted device. For handheld or fixed devices, the viewpoint moves with the movement of the handheld or fixed device and / or with the change of the user's position relative to the handheld or fixed device (e.g., the user moves toward, away from, up, down, right, and / or left). For devices including displays with video pass-through, the portion of the physical environment visible (e.g., displayed and / or projected) via one or more displays is based on the field of view of one or more cameras that communicate with the display, and the one or more cameras typically move with the display (e.g., for head-mounted devices, it moves with the user's head, or for handheld devices such as tablet computers or smart phones, it moves with the user's hands), because the user's viewpoint moves with the movement of the field of view of the one or more cameras (and the appearance of one or more virtual objects displayed via one or more displays is updated based on the user's viewpoint (e.g., the display position and posture of the virtual object are updated based on the movement of the user's viewpoint)). For displays with optical see-through, the portion of the physical environment that is visible via one or more displays (e.g., optically visible through one or more partially or fully transparent portions of the display generating components) is based on the user's field of view through the partially or fully transparent portions of the display generating components (e.g., moves with the user's head for a head-mounted device, or moves with the user's hand for a handheld device such as a tablet computer or smart phone) because the user's viewpoint moves as the user moves through the field of view of the partially or fully transparent portions of the display (and the appearance of one or more virtual objects is updated based on the user's viewpoint).

[0045] exist Figure 3A , the electronic device 101 includes a display 120 and multiple sensors as described above and controlled by the electronic device 101 to capture one or more images of the user or a portion of the user (e.g., one or more hands of the user, such as hand 305) when the user interacts with the electronic device 101. In some examples, the virtual objects, virtual content, and / or user interfaces illustrated and described below may also be implemented on a head-mounted display, which includes a display or display generation component that displays virtual objects, virtual content, user interfaces, or a three-dimensional environment to the user, and sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's attention (e.g., including gaze) (e.g., internal sensors facing inward toward the user's face). The figures herein illustrate a three-dimensional environment presented to a user by the electronic device 101 (e.g., and displayed by the display 120 of the electronic device 101). In some examples, the electronic device 101 may be similar to Figure 1 Device 101 or Figure 2 The device 201 in, and / or may be a head-mounted system / device and / or a projection-based system / device (including a hologram-based system / device) configured to generate and present a three-dimensional environment, such as, for example, a heads-up display (HUD), a head-mounted display (HMD), a window with integrated display capabilities, a display formed as a lens designed to be placed on a human eye (e.g., similar to a contact lens).

[0046] like Figure 3A , the electronic device 101 captures (e.g., using external image sensors 114 b and 114 c) one or more images of a physical environment 304 surrounding the electronic device 101, including one or more objects (e.g., table 302, paper 306, and pen 308) in the physical environment 304 surrounding the electronic device 101. In some examples, the electronic device 101 displays a representation of the physical environment 304 in a three-dimensional environment, or a portion of the physical environment 304 is visible via the display 120 of the electronic device 101. For example, the three-dimensional environment 300 includes a table 302, a paper 306, and a pen 308 in the physical environment 304.

[0047] In some examples, electronic device 101 receives indications from various applications running on the electronic device. When in physical environment 304, electronic device 101 operates in a first mode. In some examples, the first mode is a mode in which electronic device 101 receives indications from various applications and displays visual indications representing the indications without delay (e.g., in real time). When in physical environment 304, electronic device 101 presents the indications as visual indications 310 and 312 in three-dimensional environment 300, such as Figure 3A . Visual indication 310 is a notification received from a text messaging application on electronic device 101 or an electronic device communicatively connected to electronic device 101. Visual indication 312 is a notification received from an email application on electronic device 101 or an electronic device communicatively connected to electronic device 101. In some examples, the second electronic device is communicatively connected to electronic device 101 using wireless communication (e.g., WiFi, Bluetooth, and / or near field communication), physical communication (e.g., Ethernet or other wired communication), and / or a shared user account (e.g., a user logged into both devices). In some examples, the second electronic device can be a computer, a phone, a smart watch, a tablet computer, or other electronic device capable of sending and receiving communications. In some examples, the user can receive notifications from an application on the second electronic device, and the visual indication can be presented on both electronic device 101 and the second electronic device or on only one of the two devices. Reference is made below to Figure 5 The second electronic device is described in more detail.

[0048] In some examples, the electronic device 101 detects a change in the contextual information of the physical environment 304, which indicates that the current location of the electronic device 101 corresponds to the game area 314, such as Figure 3B In some examples, the user Figure 3A The office area 307 shown in FIG. 1 (eg, including the desk 302) is moved to Figure 3B . In some examples, the office area 307 and the gaming area 314 may be in different locations, such as in different buildings or different addresses. In some examples, the office area 307 and the gaming area 314 may be in the same location, such as in different rooms in one building. Figure 3B As shown in FIG. 3 , game area 314 includes various objects, such as building blocks 316. Game area 314 also includes person 318. Person 318 may be associated with a user (e.g., known to the user). For example, person 318 is a child of the user. In some examples, and as shown in FIG. Figure 3B As shown in , person 318 is interacting with a user of electronic device 101 .

[0049] In some examples, the electronic device 101 detects an event that satisfies one or more first criteria. In some examples, the one or more first criteria are satisfied when the electronic device 101 detects an interaction between the user and another person. In some examples, the interaction is a physical interaction (e.g., hugging someone, playing with someone, shaking hands with someone, or other physical interactions) and / or a verbal interaction (e.g., a conversation). In some examples, playing with someone can include direct and indirect play, such as assembling a model together, building a sand castle together, or playing cards together. For example, in Figure 3B, the user of the electronic device 101 interacts with the person 318 through the conversation 320 and through the building blocks 316. In some examples, the electronic device 101 detects speech to and from the person 318. For example, the electronic device 101 detects that the person 318 is speaking to the user using a microphone (e.g., the microphone 213 described above). In some examples, the electronic device 101 may use an image sensor (e.g., the image sensor 206) and / or a motion sensor (e.g., the orientation sensor 210) to determine physical interactions, such as playing with the building blocks 316 with the person 318. In some examples, the user of the electronic device 101 determines one or more first criteria for the electronic device 101 to transition from operating in the first mode to the second mode. For example, the one or more first criteria may include criteria that are met when the user interacts with a known person, when the user is at a specified location (e.g., at home, at work, and / or at other known locations), and / or during a specific time of day (e.g., prescribed work hours or other calendar-related events). In some examples, the electronic device 101 uses one or more sensors including the image sensor 206 to determine the identity of the person (e.g., whether the person is a known person). In some examples, electronic device 101 uses other sensors (eg, position sensor 204 , image sensor 206 , and / or other sensors) to determine whether an event satisfies one or more first criteria as defined by a user of electronic device 101 .

[0050] In some examples, in response to determining that an event satisfies one or more first criteria, the electronic device 101 transitions from operating in the first mode to the second mode. Alternatively, if the event does not satisfy one or more first criteria, the electronic device 101 does not transition to the second mode. For example, if the change in contextual information (e.g., from the office area 307 to the game area 314) and the resulting interaction do not satisfy one or more first criteria specified by the user of the electronic device 101 as described above, the electronic device 101 may remain in the first mode. In some examples, the second mode is a mode in which, if the electronic device 101 receives indications from various applications running on the electronic device 101, the electronic device 101 delays (e.g., forgoes) displaying a visual indication of the indication until the second mode is turned off. For example, the visual indication may be delayed for a variable amount of time depending on when a qualifying event is no longer detected (e.g., the user is no longer interacting with the person 318, such as when the second mode is turned off). Figure 3C). Alternatively or additionally, in some examples, when operating in the second mode, the electronic device 101 suppresses (e.g., delays or forgoes) providing other types of notification feedback (e.g., tactile and / or audio feedback). In response to transitioning to the second mode, the electronic device 101 displays a visual indication 322. The visual indication 322 optionally includes text indicating that the electronic device 101 has transitioned to the second mode (focus A is on). Alternatively, in some examples, the electronic device 101 may transition between modes without displaying a visual indication indicating a change in mode.

[0051] In some examples, when operating in a mode (e.g., the second mode) that includes a delay in the presentation of visual indications of indications from various applications, electronic device 101 still displays key notifications specified by a user of electronic device 101. In some examples, the user of electronic device 101 determines criteria that need to be met for electronic device 101 to display visual indications of notifications from various applications when operating in the second mode. For example, when in the second mode, notifications corresponding to communications from designated contacts (e.g., texts from designated contacts such as partners, parents, or other known contacts) are still shown. Figure 3B , electronic device 101 displays a visual indication 324 corresponding to a text message received from a designated contact. In some examples, communications from non-designated contacts (e.g., contacts not designated by the user or unknown contacts) are not considered critical and are therefore not displayed while presentation of notifications is suppressed (such as in the second mode). In some examples, a user of electronic device 101 may specify that notifications from specific applications (e.g., visual indications from a phone application) are displayed when in the second mode.

[0052] In some examples, the electronic device 101 uses one or more machine learning techniques and / or artificial intelligence implemented using hardware, firmware, software, or a combination thereof to determine critical notifications. For example, the electronic device 101 optionally uses a large language model (LLM) to "read" the content of an incoming communication item and determine the importance of the communication item (e.g., a text message, email, voicemail, or other communication item). For example, importance can be represented by a number (e.g., where the opposite ends of the numerical range represent the lowest or highest importance level, or can be represented as a state (e.g., low, medium, high, etc.). The communication items are described in more detail below. In some examples, after determining the importance of the communication item, the electronic device 101 compares the importance with an importance threshold (e.g., importance can be ranked on a scale of 1 to 10 and the importance threshold can be 7, or ranked on a scale of 0 to 1 and the importance threshold can be 0.8, or the importance threshold can indicate the division between importance states, etc.) to determine whether the communication item is important enough to be displayed when in the second mode. For example, a communication item with an importance determined to be equal to or greater than the importance threshold can be displayed in the second mode, while a communication item with an importance determined to be less than the importance threshold can be displayed in the second mode. Communication items with a certain importance value may not be displayed in the second mode. In some examples, the importance threshold may be based on the busyness of the user of the electronic device 101. For example, the importance threshold changes based on the user's busyness (e.g., for higher busyness, the importance threshold increases, and for lower busyness, the importance threshold decreases). In some examples, the electronic device 101 determines the user's busyness based on contextual information collected using one or more sensors, as described above. For example, if the user is actively interacting with the physical environment (e.g., writing, reading, talking to others), the busyness (and importance threshold) is higher than if the user is not actively interacting with the physical environment (e.g., watching TV, scrolling social media, etc.) (e.g., passively interacting). For example, when the electronic device 101 detects that the user is browsing the internet, the user may be prompted to browse the internet. Figure 3B 316 and interacting with person 318 through building block 316 is higher than the importance threshold when electronic device 101 detects that the user is consuming content, watching TV, listening to music, browsing the web, etc. Alternatively or additionally, in some examples, the importance threshold is based on the mode that electronic device 101 is currently in. For example, when in a Figure 3B When in the second mode (e.g., a mode indicating an increased level of concentration) shown in FIG. , the electronic device 101 has a higher level of concentration than in FIG. Figure 3A The higher importance threshold is shown in the first mode.

[0053] For example, in Figure 3B, the electronic device 101 displays the visual indication 324 because the electronic device 101 determines (eg, using the LLM as described above) that the importance level of the content of the communication item represented by the indication 324 is greater than the importance threshold.

[0054] In some examples, electronic device 101 uses machine learning and / or artificial intelligence (such as LLM) to determine an opportunistic moment to display an indication corresponding to a communication item. For example, when electronic device 101 determines that it is an appropriate time to receive a notification, electronic device 101 displays a visual indication corresponding to a communication item (e.g., both critical and / or non-critical communication items). For example, electronic device 101 uses the user's busyness to determine a less busy time to display a visual indication. In some examples, displaying notifications at appropriate times allows the user to effectively view notifications without being distracted.

[0055] In some examples, electronic device 101 detects a change in contextual information of the physical environment. For example, an event that triggers a mode change from a first mode to a second mode is removed (e.g., electronic device 101 detects movement to a location that no longer satisfies one or more first criteria, or determines that an event no longer satisfies one or more first criteria). For example, the user of electronic device 101 is no longer connected to Figure 3C 318 in the game area 314 because the current location of the electronic device 101 changes from the game area 314 back to the office area 307. In some examples, the trigger event can end when the person 318 stops interacting with the user of the electronic device 101 (for example, the electronic device 101 detects that the person 318 is no longer talking to or physically contacting the user of the electronic device 101). In some examples, the electronic device 101 transitions from the second mode back to operating in the first mode in response to no longer detecting the trigger event. Alternatively, in some examples, the electronic device 101 transitions from operating in the second mode to a different third mode in response to no longer detecting the trigger event for the second mode. For example, the electronic device 101 optionally detects different events that meet the criteria necessary for the third operating mode, and then transitions to the third mode (from the second mode or the first mode), which is described below with reference to FIG. 4A to FIG. 4C In some examples, electronic device 101 displays visual indication 326 in response to detecting a change in context information indicating that a triggering event no longer exists, such as Figure 3C In some examples, the electronic device 101 changes from operating in the second mode to the first mode without displaying a visual indication.

[0056] In some examples, transitioning back to operating in the first mode (e.g., from the second mode) includes presenting a visual indication that was suppressed when in the second mode, such as Figure 3DFor example, when operating in the second mode, the electronic device 101 displays the following information in response to receiving one or more indications (e.g., 2 text messages) from a messaging application running on the electronic device 101 (or a second electronic device communicatively connected to the electronic device 101). Figure 3D When operating in the second mode, the electronic device 101 also displays a visual indication 330 in the three-dimensional environment 300 in response to receiving one or more indications (e.g., three emails) from an email application running on the electronic device 101 (or a second electronic device communicatively connected to the electronic device 101), such as Figure 3D as shown in .

[0057] FIG. 4A to FIG. 4C Examples of a second event satisfying one or more criteria and a resulting transition of an electronic device from operating in a first mode to a third mode according to some examples of the present disclosure are illustrated. FIG. 4A to FIG. 4C To illustrate the process described below, including Figure 7 Process 700.

[0058] Figure 4A The electronic device 101 is illustrated as displaying a three-dimensional environment 400 via the display 120 from the perspective of a user of the electronic device 101 (eg, facing a staircase 406 in a room where the electronic device 101 is located). In some examples, the three-dimensional environment 400 has a reference Figure 3A One or more characteristics of the three-dimensional environment 300 described. In some examples, and as Figure 4A As shown in , electronic device 101 captures one or more images of a physical environment 404 around electronic device 101, including one or more objects (e.g., stairs 406) in the physical environment 404 around electronic device 101. In some examples, physical environment 404 includes Figure 3A One or more characteristics of the physical environment 304 described in .

[0059] In some examples, as previously referenced above FIG. 3A to FIG. 3C As described, the electronic device 101 operates in the first mode, and the user is not actively located on the stairs 406. Figure 4A In some examples, electronic device 101 detects a change in contextual information of physical environment 404, which indicates that the current position of electronic device 101 is above stair 406 (e.g., the user of electronic device 101 is located on stair 406 in physical environment 404), as shown in FIG. Figure 4B Thus, the electronic device 101 optionally transitions from operating in the first mode to a third mode (e.g., different from the second mode discussed previously above). In some examples, and as Figure 4BAs shown in , the electronic device 101 displays a visual indication 408 indicating that the electronic device 101 is operating in the third mode. In some examples, the third mode is a mode in which, if the electronic device 101 receives an indication from various applications running on the electronic device 101, the electronic device 101 delays (e.g., abandons) displaying a visual indication of the indication within a threshold amount of time. For example, the threshold amount of time may be 1 second, 1 minute, 5 minutes, 30 minutes, or 1 hour. In some examples, the user of the electronic device 101 may specify the threshold amount of time (e.g., by entering it into a mode setting user interface). In some examples, the user may specify criteria (e.g., triggering events) that need to be met to cause the electronic device 101 to transition to the third mode. For example, the third mode is reserved for activities that require short-term concentration, and the user of the electronic device 101 may specify those activities and / or criteria that need to be met. For example, an activity may include walking down (or up) stairs (e.g., walking up stairs). Figure 4B ). In some examples, when an obstacle is detected (e.g., by using image sensor 206) while electronic device 101 is operating in a navigation context (e.g., as determined by position sensor 204 and / or orientation sensor 210), the third mode is activated (e.g., automatically). For example, the obstacle may include stairs, escalators, crosswalks, train platforms, moving platforms, or other obstacles. For example, the navigation context may include walking, driving, or riding in a vehicle or other forms of transportation and navigation. In some examples, the determination that electronic device 101 is operating in a navigation context is based on data provided by a navigation application operating on electronic device 101. For example, if a user is actively navigating to a particular destination via a navigation application, then electronic device 101 is operating in a navigation context.

[0060] Figure 4C The electronic device 101 is illustrated as transitioning from the third mode back to the first mode. In some examples, after a threshold amount of time has passed, the electronic device 101 transitions back to the first mode. In some examples, and as Figure 4C As shown in , electronic device 101 switches back to the first mode after detecting a change in contextual information of the physical environment (e.g., the user of electronic device 101 has moved, such that electronic device 101 is no longer located on stair 406). In response to switching back to the first mode, electronic device 101 displays a visual indication 410 indicating that the third mode is no longer effective. In some examples, electronic device 101 displays visual indications related to notifications (indications) received from various applications that were delayed due to the effectiveness of the third mode. For example, in Figure 4C In the example, when the electronic device 101 is transformed back to the first mode, the electronic device 101 displays visual indications 412, 414, and 416. In some examples, the visual indications 412, 414, and 416 may have similar Figure 3D The visual appearance of visual indications 328 and 330 described in .

[0061] Figure 5 Examples of notifications presented on a second electronic device but not on the electronic device according to some examples of the present disclosure are illustrated. FIG. 4A to FIG. 4C To illustrate the process described below, including Figure 7 Process 700.

[0062] Figure 5 The electronic device 101 is illustrated to display a three-dimensional environment 500 from the perspective of a user of the electronic device 101 via the display 120. In some examples, the three-dimensional environment 500 has a reference Figure 3A One or more characteristics of the three-dimensional environment 300 described. In some examples, and as Figure 5 As shown in FIG. 5 , the electronic device 101 captures one or more images of the physical environment 504 around the electronic device 101, including one or more objects (e.g., table 516) in the physical environment 504 around the electronic device 101. In some examples, the physical environment 504 includes reference Figure 3A One or more characteristics of the physical environment 304 are described. Figure 5 , the physical environment 504 includes a second electronic device 512 within the field of view of the electronic device 101, and the electronic device 101 uses one or more sensors to identify the electronic device 512. In some embodiments, identifying the second electronic device 512 includes identifying that the electronic device 512 is an electronic device and identifying that the second electronic device 512 communicates with the electronic device 101. The second electronic device 512 is optionally a portable communication device, such as a mobile phone, a laptop computer, a tablet computer, a smart watch, or other devices described herein. In some examples, the second electronic device 512 is communicatively connected to the electronic device 101 via a wireless connection, a wired connection, or a shared user account. For example, a user of the electronic device 101 may have a user account associated with the electronic device 101 and the second electronic device 512, and the user logs in to the user account on the electronic device 101 and the second electronic device 512.

[0063] In some examples, in response to receiving an indication from an application, electronic device 101 does not present a visual representation for a particular application. For example, a user may specify an application and / or type of notification that electronic device 101 does not display (e.g., display as a visual representation) in three-dimensional environment 500. In some examples, electronic device 101 does not present a visual representation for a specified application when in any mode or in a particular mode (e.g., fourth mode). For example, in Figure 5In some examples, electronic device 101 defers display of a visual representation that is not displayed in three-dimensional environment 500 to a different electronic device, such as, for example, Figure 5 The second electronic device 512 is shown in FIG.

[0064] Figure 5 The electronic device 101 is illustrated as transitioning from a first mode to a fourth mode. The electronic device 101 displays a visual indication 506 indicating that the fourth mode is effective. In some examples, the user can manually change the mode. For example, the user can manually transition the electronic device 101 from the first mode to any mode (e.g., the second mode, the third mode, or the fourth mode described herein) or vice versa (e.g., from the fourth mode to the first mode). In some examples, the fourth mode is a mode in which, if the electronic device 101 receives indications from various applications running on the electronic device 101 based on a specific application, the electronic device 101 postpones the display of the indications to different devices. For example, a user of the electronic device 101 optionally indicates (e.g., by setting a user interface) that visual indications received from an email application are not displayed on the electronic device 101, but are allowed to be displayed on other devices. As Figure 5 As shown in , the visual indication 514 associated with the email application is not displayed on the electronic device 101, but is displayed on the second electronic device 512 (e.g., displayed on the touch screen of the second electronic device 512). The electronic device 101 optionally continues to display the visual indications 508 and 510 associated with the phone application and the messaging application, respectively, while giving up displaying the visual indication associated with the email application, such as Figure 5 as shown in .

[0065] FIG. 6A to FIG. 6C An example is given of a summary of notifications received when the electronic device 101 is in the first mode and the electronic device 101 is in the second mode. FIG. 6A to FIG. 6C In the embodiment, electronic device 101 displays a three-dimensional environment 600 from the perspective of a user of electronic device 101 via display 120. In some examples, three-dimensional environment 600 has reference Figure 3A One or more characteristics of three-dimensional environment 300 are described.

[0066] Fig. 6A An example is illustrated in which the electronic device 101 presents a notification when in the first mode. In some examples, the above is described in more detail and in Figure 3A The first mode is shown in Fig. 6AIn , the electronic device 101 captures (e.g., using external image sensors 114b and 114c) one or more images of the physical environment 604 surrounding the electronic device 101. Fig. 6A , the physical environment 604 corresponds to the reference Figure 4C In some examples, the electronic device 101 receives instructions from various applications without delay in the first mode. In addition, in some examples, the electronic device 101 displays a preview of the content of the communication item corresponding to the instruction. For example, in Fig. 6A , electronic device 101 displays indications 612, 614, and 616. Indications 612 and 614 are indications of communication items from a messaging application. Indication 616 is an indication of a communication item from an email application. Indications 612, 614, and 616 include previews of the content of the respective communication items. In some examples, the previews of the content of the respective communication items are direct copies of a portion of the content of the respective communication items. For example, indications 612 and 614 include previews of the respective text messages, and indication 616 includes a preview of the respective email, such as Fig. 6A as shown in .

[0067] Figure 6B An example is illustrated in which the electronic device 101 receives a notification when in the second mode, and Figure 6C An example is illustrated in which the electronic device 101 then displays an indication corresponding to the notification after re-entering the first mode. Figure 6B , the electronic device 101 is in the second mode, as illustrated by visual indication 626a. In some examples, indication 626a has a reference Figure 3B One or more characteristics of the described indication 322. In some examples, reference Figure 3B The second mode is described in more detail. Figure 6B In some examples, and as described above, the electronic device 101 delays presentation of the indication when in the second mode. For example, in Figure 6B In the example, the electronic device receives a text message from Jessica and an email from Paula (e.g., Fig. 6A (Exemplified by instructions 612, 614, and 616 shown in ).

[0068] In some examples, electronic device 101 transitions from the first mode to the second mode because electronic device 101 detects an event that satisfies one or more first criteria. For example, electronic device 101 detects a change in contextual information of the physical environment (e.g., electronic device 101 detects that electronic device 101 changes from a first mode to a second mode). Fig. 6A The living room shown in Figure 6BIn some examples, the office area 607 has a reference Figure 3A One or more characteristics of the office area 307 described. In some examples, when the electronic device 101 detects that the user of the electronic device 101 is engaged in an activity, the one or more first criteria are met. For example, Figure 6B In the example, the electronic device 101 detects that the user is working (eg, writing).

[0069] exist Figure 6C In the embodiment, the electronic device 101 transitions back to the first mode, as indicated by the visual indication 626b, which has Figure 3C In some examples, electronic device 101 detects that one or more first criteria are no longer satisfied (or one or more second criteria are satisfied). For example, electronic device 101 detects a change in contextual information of the physical environment (e.g., electronic device 101 detects that electronic device 101 has changed from Figure 6B Office area 607 to Figure 6C In response to changing the mode from the second mode to the first mode, the electronic device 101 displays indications 622 and 624, such as Figure 6C In some examples, indications 622 and 624 correspond to Fig. 6A Indications 612, 614 and 616 are shown in FIG.

[0070] In some examples, in response to detecting a mode change from the second mode back to the first mode, the electronic device 101 summarizes one or more communication items received when in the second mode (e.g., corresponding to the communication items of indications 612, 614, and 616). Alternatively, in some examples, the electronic device 101 summarizes one or more communication items received when in the second mode before detecting a mode change from the second mode back to the first mode. In some examples, the electronic device 101 displays the summary after detecting a change in mode. In some examples, the electronic device 101 classifies the indications according to the origin application (e.g., classifying all communications from a specific application into one element of the summary) and / or according to the sender (e.g., classifying all communications from a specific sender into one element of the summary). In some examples, the electronic device 101 includes all communication items from the corresponding sender (e.g., communication items from different applications (e.g., email, phone, text, voicemail, or other communication items)) into one summary indication. For example, indication 622 summarizes communication items corresponding to indications 612 and 614 from the messaging application and / or from sender Jessica, and indication 624 summarizes communication items corresponding to indication 616 from the email application and / or from sender Paula, as shown in FIG. Figure 6C As shown in Figure 6C , electronic device 101 displays indication 630 in indication 622 and indication 632 in indication 624. Indication 630 includes a representation of the sender of the message (e.g., Jessica's "J" icon) and a representation of what application (e.g., a text messaging application) the message in the summary represented by indication 622 originated from (e.g., a message icon). Figure 6C , the electronic device 101 displays an indication 632, which is a different example of an indication of a communication item representing a summary (eg, indicated by indication 624). Indication 632 includes an icon representing an application associated with the content summarized by indication 624 (eg, a mail icon).

[0071] In some examples, the electronic device uses one or more artificial intelligence methods and / or machine learning models to create a summary of the communication item. For example, electronic device 101 uses a large language model (LLM) to summarize the content of the communication item. In some examples, the LLM is stored on electronic device 101, on a server (e.g., a cloud server), or on the Internet in communication with electronic device 101. In some examples, the summary of the communication item summarizes the content of the communication item. In some examples, the summary does not include a direct copy of (part or all of) the text and / or images in the content of the communication item. For example, the summary is not a verbatim version of the content of the communication item. Figure 6CIn the embodiment, indications 622 and 624 include summaries of indications 612 to 616 that are not verbatim copies of the content in indications 612 to 616 (eg, paraphrases, etc.).

[0072] In some examples, although Figure 6C An indication of a communication item including a summary of the content of the communication item in response to the electronic device 101 transitioning from the second mode to the first mode is illustrated, but the indication of the communication item including a summary of the content of the communication item is not limited to the first mode or the second mode. For example, the electronic device 101 displays an indication of the communication item, which includes a summary of the content of the communication item in the first mode after transitioning from the third mode. In some examples, providing a summary of previously suppressed notifications allows the user to quickly and efficiently view the notifications, thereby reducing the number of inputs to the electronic device and / or improving the user experience by enabling the user to consume notifications in an efficient manner without having to read each notification / communication item in the notification / communication item in full.

[0073] Figure 7 An example process is illustrated of how an electronic device detects an event and changes mode as a result of the event. In some examples, process 700 begins at an electronic device in communication with one or more displays and one or more input devices. In some examples, the electronic device is optionally similar to or corresponding to Figure 2 The head mounted display of the device 201. Figure 7 As shown in , in some examples, when in the first mode, an electronic device such as electronic device 101 detects (702a) an event using one or more input devices including an optical sensor, such as using a sensor such as microphone 213, image sensor 206, and / or orientation sensor 210 to detect contextual information. For example, the event is a person (such as Figure 3C Interaction between people in the network (318).

[0074] In some examples, based on a determination that an event satisfies one or more first criteria (the one or more first criteria including a first criterion satisfied when an interaction with a person is detected), the electronic device transitions (702b) from the first mode to the second mode, such as with Figure 3B As shown in the display of visual indication 322.

[0075] In some examples, while in the second mode (702c), the electronic device receives (602d) a notification from an application. For example, the electronic device 101 receives a notification from a messaging application indicating that the user has received a text message from a different user. In some examples, if the user is not in the second mode, the notification will be displayed as a visual indication, such as Figure 3A Visual indications 310 and 312 in FIG.

[0076] In some examples, when in the second mode (702c), based on a determination that one or more second criteria are met, the electronic device (702e) forgoes presenting a notification (e.g., a tactile sensation, an audio message, a visual notification) from the application on the electronic device. Figure 3B As shown in , only notifications (e.g., visual indications 324) that do not meet one or more second criteria are displayed on electronic device 101. In some examples, electronic device 101 uses machine learning and / or artificial intelligence (e.g., LLM) to determine notifications that do not meet one or more second criteria. For example, as described above, notifications with an importance greater than an importance threshold are displayed in the second mode.

[0077] In some examples, after the electronic device 101 transitions back to the first mode (e.g., after one or more first criteria are no longer met), the electronic device 101 optionally displays one or more indications summarizing the content of notifications received while the electronic device 101 was in the second mode, such as FIG. 6A to FIG. 6C Described in .

[0078] It should be understood that process 700 is an example, and more, fewer, or different operations may be performed in the same or different order. In addition, the operations in the above process 700 are optionally performed by running an information processing device such as a general-purpose processor (e.g., as described in relation to Figure 2 The one or more functional modules in the dedicated chip and / or Figure 2 other components to achieve this.

[0079] Therefore, according to the above, some examples of the present disclosure relate to a method, the method comprising: at an electronic device that communicates with one or more displays and one or more input devices: when in a first mode, using one or more input devices including an optical sensor to detect an event; based on a determination that the event satisfies one or more first criteria, transitioning the electronic device from the first mode to the second mode, the one or more first criteria including a first criterion that is satisfied when the event corresponds to an interaction with a person; when in the second mode: receiving a notification from an application; and based on a determination that one or more second criteria are satisfied, abandoning presentation of the notification from the application on the electronic device. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further comprises: based on a determination that the event does not satisfy one or more first criteria, abandoning the transition of the electronic device from the first mode to the second mode. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further comprises: when in the second mode, based on a determination that the event has ended, transitioning the electronic device from the second mode to the first mode. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further comprises: in response to transitioning the electronic device from the second mode to the first mode, presenting one or more notifications that were received and not presented when in the second mode. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: detecting an event using one or more input devices including an optical sensor, which includes determining the identity of the person. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: detecting an event using one or more input devices including an optical sensor, which includes detecting a physical interaction with a person. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: detecting an event, which also includes detecting audio information using one or more input devices. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: detecting audio information, which also includes detecting speech to or from a person. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: detecting an event, which also includes detecting location information using one or more input devices. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: when in the second mode, presenting a notification from an application on the electronic device after a threshold amount of time based on a determination that one or more third criteria are met.In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: when in the second mode, presenting a notification from the application on a second electronic device communicatively coupled to the electronic device and abandoning presenting the notification on the electronic device based on a determination that one or more fourth criteria are satisfied. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: when in the second mode, presenting a notification from the application on the electronic device based on a determination that one or more second criteria are not satisfied. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the aforementioned presenting of the notification from the application also includes the aforementioned presenting of the notification when the electronic device is in the second mode. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the aforementioned presenting of the notification from the application also includes presenting the notification after a threshold amount of time. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the aforementioned presenting of the notification includes causing the notification to be presented on a second electronic device communicatively coupled to the electronic device and the aforementioned presenting of the notification on the electronic device. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the one or more first criteria include criteria that are satisfied when the notification corresponds to a communication from a non-specified contact. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the method further includes: when in the first mode, detecting a second event using one or more input devices including an optical sensor; and transitioning the electronic device from the first mode to the third mode based on a determination that the second event satisfies one or more fifth criteria (the one or more fifth criteria including criteria satisfied when an obstacle is detected in a navigation context). In addition to or in lieu of one or more of the examples disclosed above, in some examples, presenting one or more notifications received and not presented when in the second mode also includes presenting one or more summaries corresponding to the one or more notifications. In addition to or in lieu of one or more of the examples disclosed above, in some examples, the one or more second criteria include an importance criterion satisfied when the notification has an importance less than the notification criterion.

[0080] Some examples of the present disclosure relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for executing any of the above methods.

[0081] Some examples of the present disclosure relate to a non-transitory computer-readable storage medium that stores one or more programs that include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the above methods.

[0082] Some examples of the present disclosure relate to an electronic device comprising one or more processors, a memory, and a component for performing any of the above methods.

[0083] Some examples of the present disclosure relate to an information processing apparatus used in an electronic device, the information processing apparatus including a component for executing any one of the above methods.

[0084] The present disclosure contemplates that, in some instances, the data utilized may include personal information data that uniquely identifies or can be used to contact or locate a particular person. Such personal information data may include demographic data, content consumption activity, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information. Specifically, as described herein, one aspect of the present disclosure is to view the content of a communication item and summarize the content.

[0085] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to change the mode of the electronic device. For example, the electronic device changes the mode based on changes to the user's environment and / or interaction with the environment.

[0086] The disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information data will comply with sound privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easy for users to access and should be updated as changes in the collection and / or use of data occur. Personal information from users should be collected for legal and reasonable entity purposes and should not be shared or sold outside these legal purposes. In addition, such collection / sharing should be carried out after receiving the informed consent of the user. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data, and ensure that other entities with access to personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adapted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including considerations of special jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.

[0087] Regardless of the foregoing, the present disclosure also contemplates examples where users selectively block the use or access of personal information data. That is, the present disclosure contemplates hardware elements and / or software elements to prevent or block access to such personal information data. For example, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" to participate in the collection of personal information data during or at any time after registration for a service. In another example, a user may choose not to enable the recording of personal information data in a particular application (e.g., a first application and / or a second application). In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed after the collection is initiated, and then the user may be reminded again just before the device accesses the personal information data.

[0088] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting the collection of data and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0089] For the purpose of explanation, the foregoing description is described by reference to specific examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Based on the above teachings, many modifications and variations are possible. The examples are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so as to enable other technical personnel in the field to best use the present disclosure and the various described examples with various modifications suitable for the specific purpose contemplated.

Claims

1. A method, comprising: At an electronic device in communication with one or more displays and one or more input devices: when in the first mode, detecting an event using the one or more input devices including an optical sensor; transitioning the electronic device from the first mode to a second mode based on a determination that the event satisfies one or more first criteria, the one or more first criteria including a first criterion that is satisfied when the event corresponds to an interaction with a person; When in the second mode: Receive notifications from apps; as well as Based on a determination that one or more second criteria are met, presenting the notification from the application on the electronic device is foregone.

2. The method according to claim 1, further comprising: Based on a determination that the event does not satisfy the one or more first criteria, transitioning the electronic device from the first mode to the second mode is abandoned.

3. The method according to claim 1, further comprising: While in the second mode, the electronic device is transitioned from the second mode to the first mode based on a determination that the event has ended.

4. The method according to claim 3, further comprising: In response to transitioning the electronic device from the second mode to the first mode, presenting one or more notifications that were received and not presented while in the second mode. 5 . The method of claim 4 , wherein presenting the one or more notifications that were received and not presented while in the second mode further comprises presenting one or more summaries corresponding to the one or more notifications. The method of claim 1 , wherein the one or more second criteria include an importance criterion that is satisfied when the notification has an importance that is less than an importance criterion. 7 . The method of claim 1 , wherein detecting the event using the one or more input devices including the optical sensor comprises determining an identity of the person or detecting a physical interaction with the person.

8. The method of claim 1, wherein detecting the event further comprises detecting audio information using the one or more input devices.

9. The method of claim 1, wherein detecting audio information further comprises detecting speech to or from the person.

10. The method of claim 1, wherein detecting the event further comprises detecting location information using the one or more input devices.

11. The method according to claim 1, further comprising: While in the second mode, the notification from the application is presented on the electronic device after a threshold amount of time based on a determination that one or more third criteria are satisfied.

12. The method according to claim 1, further comprising: While in the second mode, based on a determination that one or more fourth criteria are satisfied, presenting the notification from the application on a second electronic device communicatively coupled to the electronic device and forgoing presenting the notification on the electronic device.

13. The method according to claim 1, further comprising: While in the second mode, based on the determination that the one or more second criteria are not satisfied, presenting the notification from the application on the electronic device.

14. The method of claim 1, wherein forgoing presenting the notification from the application further comprises forgoing presenting the notification while the electronic device is in the second mode.

15. The method of claim 1, wherein forgoing presenting the notification from the application further comprises presenting the notification after a threshold amount of time.

16. The method of claim 1, wherein forgoing presentation of the notification comprises causing the notification to be presented on a second electronic device communicatively coupled to the electronic device and forgoing presentation of the notification on the electronic device.

17. The method of claim 1, wherein the one or more second criteria include a criterion that is satisfied when the notification corresponds to a communication from a non-designated contact.

18. The method according to claim 1, further comprising: while in the first mode, detecting a second event using the one or more input devices including the optical sensor; as well as The electronic device is transitioned from the first mode to a third mode based on a determination that the second event satisfies one or more fifth criteria, the one or more fifth criteria including criteria that are satisfied when an obstacle is detected while in a navigation context.

19. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform a method according to any one of claims 1 to 18.

20. An electronic device, comprising: one or more processors; Memory; as well as One or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 18.

Images