Method for displaying user interface elements related to media content

Abstract

In some embodiments, a computer system displays subtitles for a media item at different depths as a function of the depth of the portion of the media item over which the subtitles are displayed. In some embodiments, the computer system displays user interface elements including information associated with the media item at different locations relative to the media item according to the attention of the user. In some embodiments, the computer system displays a user interface element including information associated with the media item in different visual appearances according to visual characteristics of the portion of the media item over which the user interface element is displayed.

Description

[0001] This application is a divisional application based on the Chinese invention patent application with application date of January 10, 2023, application number 2023800264721, and invention name “Method for displaying user interface elements related to media content”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 63 / 266,625, filed on January 10, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] The present invention generally relates to computer systems that provide computer-generated experiences, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via displays. Background Art

[0005] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays) for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Exemplary virtual elements include virtual objects such as digital images, videos, text, icons, and control elements (such as buttons and other graphics). Summary of the invention

[0006] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve desired results in augmented reality environments, and systems that make virtual object manipulation complex, cumbersome, and error-prone can impose a huge cognitive burden on users and detract from the experience of virtual / augmented reality environments. In addition, these methods take longer than necessary, wasting energy on the computer system. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, there is a need for computer systems with improved methods and interfaces to provide computer-generated experiences to users, so that the user's interaction with the computer system is more efficient and more intuitive to the user. Such methods and interfaces optionally supplement or replace conventional methods for providing extended reality experiences to users. Such methods and interfaces reduce the amount, degree and / or nature of inputs from the user by helping the user understand the connection between the inputs provided and the device's response to those inputs, thereby forming a more effective human-computer interface.

[0008] The above-mentioned defects and other problems associated with the user interface of the computer system are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or a head-mounted device). In some embodiments, the computer system has a touch pad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, in addition to displaying a generating component, the computer system also has one or more output devices, which include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI through contact and gestures of a stylus and / or finger on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body (as captured by a camera and other motion sensors), and / or voice input (as captured by one or more audio input devices). In some embodiments, the functions performed by interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, testing support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a transient and / or non-transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] There is a need for electronic devices with improved methods and interfaces for interacting with content in a three-dimensional environment. Such methods and interfaces can supplement or replace conventional methods for interacting with content in a three-dimensional environment. Such methods and interfaces reduce the amount, degree, and / or nature of input from a user and produce a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges.

[0010] In some embodiments, the computer system displays the subtitles for the media item at different depths according to the depth of the part of the media item on which the subtitles are displayed. In some embodiments, the computer system displays the user interface elements including the information associated with the media item at different positions relative to the media item according to the user's attention. In some embodiments, the computer system displays the user interface elements including the information associated with the media item with different visual appearances according to the visual characteristics of the part of the media item on which the user interface elements are displayed.

[0011] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not comprehensive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art based on the drawings, the specification, and the claims. In addition, it should be noted that the language used in this specification is selected in principle for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0013] Figure 1 is a block diagram illustrating an operating environment of a computer system for providing an XR experience according to some embodiments.

[0014] Figure 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience according to some embodiments.

[0015] Figure 3 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of an XR experience to a user according to some embodiments.

[0016] Figure 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture input from a user according to some embodiments.

[0017] Figure 5 is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input of a user according to some embodiments.

[0018] Figure 6 is a flow chart illustrating a flash-assisted gaze tracking pipeline according to some embodiments.

[0019] 7A to 7E An example is shown of how an electronic device according to some embodiments of the present disclosure displays video subtitles for a media item having content at different depths.

[0020] FIG. 8A to FIG. 8I is a flowchart illustrating an exemplary method of displaying video subtitles for a media item having content at different depths according to some embodiments.

[0021] 9A to 9E An example is shown of how an electronic device according to some embodiments of the present disclosure can display video subtitles for a media item based on the attention of a user of the electronic device.

[0022] FIG. 10A to FIG. 10G is a flowchart illustrating a method for displaying video subtitles for a media item based on the attention of a user of an electronic device according to some embodiments.

[0023] FIG. 11A to FIG. 11C Examples are shown of how electronic devices according to some embodiments of the present disclosure may display video subtitles with different visual appearances.

[0024] FIG. 12A to FIG. 12F is a flowchart illustrating a method of displaying video subtitles with different visual appearances according to some embodiments. DETAILED DESCRIPTION

[0025] According to some embodiments, the present disclosure relates to a user interface for providing an extended reality (XR) experience to a user.

[0026] The systems, methods, and GUIs described herein provide improved ways for electronic devices to facilitate interaction with and manipulation of objects in a three-dimensional environment.

[0027] In some embodiments, the computer system displays a representation of a media item with content at different depths. In some embodiments, the computer system covers a portion of the representation of the media item and displays a user interface element comprising a caption for the media item. If the portion of the representation of the media item is being presented at a first depth, the user interface element is displayed at a first distance from the user's viewpoint. If the portion of the representation of the media item is being presented at a second depth, the user interface element is displayed at a second distance from the user's viewpoint.

[0028] In some embodiments, the computer system displays a representation of the media item. In some embodiments, the computer system displays a user interface element including information associated with the media item at a first position corresponding to a first portion of the representation of the media item relative to the representation of the media item. In response to detecting that the user's attention has changed from being directed to the first portion of the representation of the media item to being directed to the second portion, the computer system optionally displays the user interface element at a second position corresponding to the second portion of the representation of the media item relative to the representation of the media item.

[0029] In some embodiments, a computer system displays a representation of a media item. In some embodiments, a computer system displays a user interface element including information associated with the media item overlying a portion of the representation of the media item, and the user interface element has a first visual appearance based on one or more visual characteristics of the portion of the representation of the media item. In some embodiments, as the playback positioning of the media item changes and one or more visual characteristics of the portion of the representation of the media item on which the user interface element is displayed change, the computer system displays the user interface element with a second visual appearance that is different from the first visual appearance.

[0030] Figures 1 to 6 A description of an example computer system for providing an XR experience to a user (such as described below with respect to methods 800 , 1000 , and / or 1200 ) is provided. 7A to 7E An example is shown of how an electronic device according to some embodiments of the present disclosure displays video subtitles for a media item having content at different depths. FIG. 8A to FIG. 8I is a flowchart illustrating an exemplary method of displaying video subtitles for a media item having content at different depths according to some embodiments. 7A to 7E The user interface in FIG. 8A to FIG. 8I process. 9A to 9E An example is shown of how an electronic device according to some embodiments of the present disclosure can display video subtitles for a media item based on the attention of a user of the electronic device. FIG. 10A to FIG. 10G is a flowchart illustrating a method for displaying video subtitles for a media item based on the attention of a user of an electronic device according to some embodiments. 9A to 9E The user interface in FIG. 10A to FIG. 10G process. FIG. 11A to FIG. 11C Examples are shown of how electronic devices according to some embodiments of the present disclosure may display video subtitles with different visual appearances. FIG. 12A to FIG. 12F is a flowchart illustrating a method of displaying video subtitles with different visual appearances according to some embodiments. FIG. 11A to FIG. 11C The user interface in FIG. 12A to FIG. 12F process.

[0031] The processes described below enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation without further user input when a set of conditions have been met, improving privacy and / or security, and / or additional techniques. These techniques also reduce power usage and extend the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0032] In addition, in the method described herein where one or more steps depend on one or more conditions being met, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions for determining the steps in the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the ordinary technician will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on one or more conditions being met can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether the possible situation has been met without explicitly repeating the steps of the method until all conditions for determining the steps in the method have been met. It will also be understood by ordinary technicians in the art that, similar to the method with the contingent step, the system or computer-readable storage medium can repeat the steps of the method as many times as needed to ensure that all the contingent steps have been performed.

[0033] In some embodiments, such as Figure 1As shown in , an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a household appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head mounted device or a handheld device).

[0034] In describing an XR experience, various terms are used to distinguishably refer to several related but distinct environments that a user can sense and / or with which the user can interact (e.g., using inputs detected by the computer system 101 generating the XR experience, which inputs cause the computer system generating the XR experience to generate audio, visual, and / or tactile feedback corresponding to the various inputs provided to the computer system 101). The following is a subset of these terms:

[0035] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0036] Extended Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In XR, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. For example, an XR system may detect a person's head turn, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to the features of virtual objects in the XR environment may be made in response to a representation of physical movement (e.g., a voice command). People can sense and / or interact with XR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, people can sense and / or interact with audio objects, which create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some XR environments, a person can sense and / or interact only with an audio object.

[0037] Examples of XR include virtual reality and mixed reality.

[0038] Virtual Reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0039] Mixed Reality: In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between, but not including, a fully physical environment at one end and a virtual reality environment at the other end. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render MR environments can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment, or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0040] Examples of mixed reality include augmented reality and augmented virtuality.

[0041] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. A person uses the system to indirectly view the physical environment via an image or video of the physical environment and perceives virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, such as as a hologram or on a physical surface, so that a person using the system perceives virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not real version of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating a portion thereof or blurring a portion thereof.

[0042] Augmented Virtual: An augmented virtual (AV) environment refers to a simulated environment in which a virtual environment or a computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0043] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same position and / or location in a user's viewpoint, the virtual object is viewpoint-locked even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the forward direction of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); thus, without moving the user's head, the user's viewpoint remains fixed even when the user's gaze shifts. In embodiments where the computer system has a display generation component (e.g., a display screen) that is repositionable relative to the user's head, the user's viewpoint is an augmented reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object that is displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or location of the viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object."

[0044] Environment-locked visual objects: A virtual object is environment-locked (alternatively, "world-locked") when a computer system displays a virtual object at a position and / or location in a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) a location and / or object in a three-dimensional environment (e.g., a physical environment or a virtual environment). As the user's viewpoint moves, the location and / or objects in the environment change relative to the user's viewpoint, which causes the environment-locked virtual object to be displayed at a different location and / or location in the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) such that the tree is now to the left of center in the user's viewpoint (e.g., the tree location in the user's viewpoint shifts), the environment-locked virtual object locked to the tree is displayed to the left of center in the user's viewpoint. In other words, the position and / or orientation of the virtual object that is locked to the environment in the user's viewpoint depends on the position and / or orientation of the object and / or the position to which the virtual object is locked in the environment. In some embodiments, the computer system uses a stationary reference system (e.g., a coordinate system anchored to a fixed position and / or object in the physical environment) to determine the positioning of the virtual object that is locked to the user's viewpoint. The virtual object that is locked to the environment can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object), or can be locked to a movable part of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot of the user) so that the virtual object moves as the viewpoint or the part of the environment moves to maintain a fixed relationship between the virtual object and the part of the environment.

[0045] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits an inertial following behavior that reduces or delays the movement of the environment-locked or viewpoint-locked virtual object relative to the movement of the reference point followed by the virtual object. In some embodiments, when exhibiting inertial following behavior, when detecting the movement of a reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 cm and 300 cm from the viewpoint) that the virtual object is following, the computer system intentionally delays the movement of the virtual object. For example, when the reference point (e.g., the portion of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits inertial following behavior, the device ignores small amounts of movement of the reference point (e.g., ignoring movements of the reference point below a threshold movement amount, such as moving 0 to 5 degrees or moving 0 to 50 cm). For example, when a reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and when the reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., a “lazy follow” threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some embodiments, maintaining a substantially fixed position of the virtual object relative to the reference point includes displaying the virtual object within a threshold distance (e.g., 1 cm, 2 cm, 3 cm, 5 cm, 15 cm, 20 cm, 50 cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the position of the reference point).

[0046] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to become selectively opaque. The projection-based system can employ retinal projection technology that projects graphic images onto a person's retina. The projection system can also be configured to project virtual objects into a physical environment, such as as a hologram or on a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Reference is made below to Figure 2The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device that is located locally or remotely relative to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. As another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the display generation component 120 (e.g., HMD, display, projector, touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included within a housing (e.g., a physical housing) of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors, etc.), one or more input devices of the input devices 125, one or more output devices of the output devices 155, one or more sensors of the sensors 190, and / or one or more peripheral devices of the peripheral devices 195, or shares the same physical housing or support structure with one or more of the above devices.

[0047] In some embodiments, the display generation component 120 is configured to provide an XR experience (e.g., at least the visual component of the XR experience) to the user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Display generation component 120 is described in further detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0048] According to some embodiments, display generation component 120 provides an XR experience to the user when the user is virtually and / or physically present within scene 105.

[0049] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head, on his / her hand, etc.). In this way, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smart phone or tablet) configured to present XR content, and the user holds a device with a display facing the user's field of view and a camera facing the scene 105. In some embodiments, the handheld device is optionally placed in a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is an XR room, housing, or room configured to present XR content, wherein the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in the space in front of a handheld device or a tripod-mounted device may similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and responses to the XR content are displayed via the HMD. Similarly, a user interface showing interactions with XR content triggered based on movement of a handheld device or a tripod-mounted device relative to a physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)) may similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)).

[0050] Despite Figure 1 Relevant features of operating environment 100 are shown, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein.

[0051] Figure 2is a block diagram of an example of a controller 110 according to some embodiments. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), IEEE802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0052] In some embodiments, the one or more communication buses 204 include circuits that interconnect and control communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0053] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 230 and an XR experience module 240.

[0054] The operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0055] In some embodiments, the data acquisition unit 241 is configured to obtain Figure 1 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data acquisition unit 241 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0056] In some embodiments, tracking unit 242 is configured to map scene 105 and track at least display generation component 120 relative to Figure 1 105, and optionally relative to the position / location of one or more of the tracked input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand, and / or the position / location of one or more parts of the user's hand relative to the user's hand. Figure 1 The movement of the scene 105 relative to the display generation component 120 and / or relative to a coordinate system (the coordinate system is defined relative to the user's hand). Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the position or movement of the user's gaze (or more generally, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hands)) or relative to the XR content displayed via the display generation component 120. Figure 5 The eye tracking unit 243 is described in more detail.

[0057] In some embodiments, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripherals 195. To this end, in various embodiments, coordination unit 246 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0058] In some embodiments, data transfer unit 248 is configured to transfer data (e.g., presentation data, position data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data transfer unit 248 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0059] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are illustrated as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may be located in separate computing devices.

[0060] also, Figure 2 It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0061] Figure 31 is a block diagram of an example of a display generation component 120 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the display generation component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0062] In some embodiments, one or more communication buses 304 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0063] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to the user. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more XR displays 312 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. In another example, the display generation component 120 includes an XR display for each eye of the user. In some embodiments, one or more XR displays 312 can present MR and VR content. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0064] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to the user's hands and, optionally, at least a portion of the user's arms (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to a scene that the user would see in the absence of the display generating component 120 (e.g., an HMD) (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0065] The memory 320 includes a high-speed random access memory, such as a DRAM, SRAM, DDR RAM, or other random access solid-state memory device. In some embodiments, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 330 and an XR rendering module 340.

[0066] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR rendering module 340 is configured to present XR content to a user via one or more XR displays 312. To this end, in various embodiments, the XR rendering module 340 includes a data acquisition unit 342, an XR rendering unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0067] In some embodiments, the data acquisition unit 342 is configured to at least Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For the purposes described, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0068] In some embodiments, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. For such purposes, in various embodiments, the XR rendering unit 344 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0069] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which a computer-generated object can be placed to generate an extended reality) based on the media content data. For such purposes, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0070] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, position data, etc.) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For such purposes, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0071] Although the data acquisition unit 342, the XR rendering unit 344, the XR map generation unit 346, and the data transfer unit 348 are shown as residing on a single device (e.g., Figure 1 , but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0072] also, Figure 3 More as a functional description of various features that may be present in a particular embodiment, rather than a schematic diagram of the structures of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0073] Figure 4 is a schematic illustration of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1 ) is controlled by the hand tracking unit 244 ( Figure 2 ) to track the position / location of one or more parts of a user's hand and / or one or more parts of a user's hand relative to Figure 1The hand tracking device 140 can be used to monitor movement of the scene 105 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system (which is defined relative to the user's hands)). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to the head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0074] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a human user's hand 406. The image sensor 404 captures hand images with sufficient resolution so that fingers and their corresponding positions can be distinguished. The image sensor 404 typically captures images of other parts of the user's body, and also or possibly all parts of the body, and may have zoom capabilities or a dedicated sensor with increased magnification to capture images of the hand with a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or is used as an image sensor to capture the physical environment of the scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a manner that the field of view of the image sensor 404 or a portion thereof is used to define an interaction space, in which the hand movements captured by the image sensor are considered as input to the controller 110.

[0075] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D mapping data (and in addition, possible color image data) to the controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided to an application running on the controller via an application program interface (API), which drives the display generation component 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving his hand 406 and changing his hand posture.

[0076] In some embodiments, the image sensor 404 projects a speckled pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral offset of the spots in the pattern. This method is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. The method gives the depth coordinates of points in the scene at a specific distance from the image sensor 404 relative to a predetermined reference plane. In the present disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis, so that the depth coordinates of points in the scene correspond to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods based on a single or multiple cameras or other types of sensors, such as stereo imaging or time-of-flight measurement.

[0077] In some embodiments, the hand tracking device 140 captures and processes a time series of depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or the processor in the controller 110 processes the 3D mapping data to extract image patch descriptors of the hand in these depth maps. The software can match these descriptors with image patch descriptors stored in the database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and finger tips.

[0078] The software can also analyze the trajectory of the hand and / or finger over multiple frames in the sequence to recognize gestures. The pose estimation function described herein can be alternated with the motion tracking function so that the image block-based pose estimation is performed only once every two (or more) frames, and the tracking is used to find the changes in pose that occur on the remaining frames. The pose, motion, and gesture information is provided to the application running on the controller 110 via the above-mentioned API. The program can, for example, move and modify the image presented on the display generation component 120 in response to the pose and / or gesture information, or perform other functions.

[0079] In some embodiments, gestures include air gestures. An air gesture is a gesture detected without the user touching an input element that is part of a device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) (or independent of an input element that is part of a device) and based on detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of a user's finger relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0080] In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with an XR environment (e.g., a virtual or mixed reality environment) performed by movement of a user's fingers relative to other fingers or parts of the user's hand. In some embodiments, an air gesture is a gesture detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and based on detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture including a predetermined speed or rotation amount of a part of the user's body)).

[0081] In some embodiments where the input gesture is an in-air gesture (e.g., in the absence of physical contact with an input device that provides information to the computer system about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Thus, in embodiments involving in-air gestures, for example, the input gesture is combined (e.g., simultaneously) with movement of the user's fingers and / or hand to detect attention (e.g., gaze) toward a user interface element to perform a pinch and / or tap input, as described below.

[0082] In some embodiments, an input gesture pointing to a user interface object is performed directly or indirectly with reference to the user interface object. For example, the user input is performed directly on the user interface object according to performing input with the user's hand at a location corresponding to the location of the user interface object in the three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some embodiments, when the user's attention (e.g., gaze) to the user interface object is detected, the input gesture is performed indirectly on the user interface object according to the location of the user's hand not being at the location corresponding to the location of the user interface object in the three-dimensional environment while the user performs the input gesture. For example, for a direct input gesture, the user can guide the user's input to the user interface object by initiating a gesture at or near a location corresponding to the display location of the user interface object (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge of the option or the center portion of the option). For an indirect input gesture, the user can guide the user's input to the user interface object by focusing on the user interface object (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location that does not correspond to the display location of the user interface object).

[0083] In some embodiments, according to some embodiments, the input gestures (e.g., air gestures) used in various examples and embodiments described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0084] In some embodiments, the pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes movement of two or more fingers of a hand to contact each other, that is, optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact with each other. A long pinch gesture as an air gesture includes movement of two or more fingers of a hand in contact with each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact with each other is detected. For example, a long pinch gesture includes the user maintaining a pinch gesture (e.g., in which two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture as an air gesture includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected consecutively immediately (e.g., within a predefined time period) with each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts contact between two or more fingers), and performs a second pinch input within a predefined time period after releasing the first pinch input (e.g., within 1 second or within 2 seconds).

[0085] In some embodiments, the pinch and drag gesture as an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., following) a drag input that changes the position of the user's hand from a first position (e.g., the start position of the drag) to a second position (e.g., the end position of the drag). In some embodiments, the user maintains the pinch gesture while performing the drag input, and releases the pinch gesture (e.g., opens their two or more fingers) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers to contact each other and moves the same hand to a second position in the air using the drag gesture). In some embodiments, a pinch input is performed by a first hand of a user, and a drag input is performed by a second hand of the user (e.g., while the user continues the pinch input with the user's first hand, the user's second hand moves in the air from a first position to a second position. In some embodiments, the input gesture as an air gesture includes an input performed using both hands of the user (e.g., a pinch and / or tap input). For example, the input gesture includes two (e.g., or more) pinch inputs performed in conjunction with each other (e.g., simultaneously or within a predefined time period). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) is performed using the first hand of the user, and in conjunction with the pinch input performed using the first hand, a second pinch input is performed using another hand (e.g., a second hand of the user's two hands). In some embodiments, movement between the two hands of the user (e.g., increasing and / or decreasing the distance or relative orientation between the two hands of the user)

[0086] In some embodiments, a tap input performed as an air gesture (e.g., pointing to a user interface element) includes movement of a user's finger toward the user interface element, movement of a user's hand toward the user interface element (optionally, extension of the user's finger toward the user interface element), downward movement of a user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of a user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture movement of the finger or hand, which is a movement of the finger or hand away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by an end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of the movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of movement of the finger or hand, and / or a reversal of the acceleration direction of the movement of the finger or hand).

[0087] In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment (optionally, no other conditions are required). In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment using one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., a dwell duration) and / or requiring the gaze to be directed to the portion of the three-dimensional environment when the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment, so that the device determines that the user's attention is directed to the portion of the three-dimensional environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0088] In some embodiments, detection of a ready state configuration of a user or a portion of a user is detected by a computer system. Detection of a ready state configuration of a hand is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein) performed by the hand. For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart to prepare for a pinch or grab gesture, or a pre-tap with one or more fingers extended and the palm facing away from the user), based on whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or based on whether the hand has moved in a particular manner (e.g., toward an area in front of the user above the user's waist and below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of a user interface responds to attention (e.g., gaze) input.

[0089] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or may alternatively be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is also stored in a memory associated with the controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although in Figure 4Controller 110 is shown in FIG. 1 , but some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand tracking device) or other device associated with the image sensor 404, for example, as a separate unit from the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing functions of the image sensor 404 may also be integrated into a computer or other computerized device to be controlled by the sensor output.

[0090] Figure 4 Also included is a schematic diagram of a depth map 410 captured by the image sensor 404 according to some embodiments. As described above, the depth map includes a matrix of pixels with corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z distance from the image sensor 404), where gray shades become darker as depth increases. The controller 110 processes these depth values ​​in order to identify and segment components of the image that have human hand features (i.e., a group of adjacent pixels). These features may include, for example, overall size, shape, and motion from frame to frame in the depth map sequence.

[0091] Figure 4 Also schematically shown is a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406 according to some embodiments. Figure 4 In the hand skeleton 414, a hand background 416 that has been segmented from the original depth map is superimposed. In some embodiments, key feature points of the hand and optionally on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, finger tips, palm center, end of the hand connected to the wrist, etc.) are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points over multiple image frames to determine the gesture performed by the hand or the current state of the hand according to some embodiments.

[0092] Figure 5 The eye tracking device 130 ( Figure 1 ). In some embodiments, the eye tracking device 130 is composed of an eye tracking unit 243 ( Figure 2) controls to track the position and movement of the user's gaze relative to the scene 105 or relative to the XR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as headphones, helmets, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for viewing by the user and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye tracking device 130 is optionally a device separate from the handheld device or the XR room. In some embodiments, the eye tracking device 130 is a head-mounted device or a part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or a display generation component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally used in conjunction with a head mounted display generation component. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally part of a non-head mounted display generation component.

[0093] In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display a frame including a left image and a right image in front of the user's eyes, thereby providing a 3D virtual view to the user. For example, the head-mounted display generation component may include a left optical lens and a right optical lens (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture a video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display, and display a virtual object on the transparent or translucent display, and the user can directly view the physical environment through the transparent or translucent display. In some embodiments, the display generation component projects the virtual object into the physical environment. The virtual object may, for example, be projected on a physical surface or projected as a hologram so that an individual using the system observes the virtual object superimposed on the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

[0094] like Figure 5As shown in , in some embodiments, the eye tracking device 130 (e.g., gaze tracking device) includes at least one eye tracking camera (e.g., an infrared (IR) or near infrared (NIR) camera), and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive the IR or NIR light reflected directly from the eyes by the light source, or alternatively can be pointed at "hot" mirrors located between the user's eyes and the display panel, which reflect the IR or NIR light from the eyes to the eye tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and transmits the gaze tracking information to the controller 110. In some embodiments, the user's two eyes are tracked separately by corresponding eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by the corresponding eye tracking camera and illumination source.

[0095] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for a specific operating environment 100, such as the 3D geometric relationships and parameters of the LED, camera, thermal mirror (if present), eye lens, and display screen. The device-specific calibration process can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration process can be an automatic calibration process or a manual calibration process. According to some embodiments, the user-specific calibration process may include an estimate of the eye parameters of a specific user, such as pupil position, foveal position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific parameters and user-specific parameters are determined for the eye tracking device 130, a flash-assisted method can be used to process the images captured by the eye tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0096] like Figure 5As shown in , the eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system, which includes at least one eye tracking camera 540 (e.g., an infrared (IR) or near infrared (NIR) camera) positioned on the side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye tracking camera 540 can be directed toward a mirror 550 (which reflects IR or NIR light from the eye 592 while allowing visible light to pass) located between the user's eye 592 and a display 510 (e.g., a left display panel or a right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) Figure 5 ), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion of Figure 5 ).

[0097] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0098] Several possible use cases for the user's current gaze direction are described below and are not intended to be limiting. As an exemplary use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content at a higher resolution in the foveal region determined according to the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller may display specific virtual content in a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 may direct an external camera used to capture the physical environment of the XR experience to focus in the determined direction. The autofocus mechanism of the external camera may then focus on an object or surface in the environment that the user is currently looking at on the display 510. As another exemplary use case, the eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of the eye lens 520 so that the virtual object that the user is currently looking at has an appropriate degree of convergence to match the convergence of the user's eye 592. The controller 110 can use the gaze tracking information to guide the eye lens 520 to adjust the focus so that nearby objects that the user is looking at appear at the correct distance.

[0099] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lenses 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)). The light source emits light (e.g., IR or NIR light) toward the user's eye 592. In some embodiments, the light sources may be arranged in a ring or circle around each of the lenses, such as Figure 5 In some embodiments, for example, eight light sources 530 (eg, LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0100] In some embodiments, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. Note that the positions and angles of the eye tracking cameras 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0101] like Figure 5 Embodiments of the gaze tracking system shown in may, for example, be used in computer generated reality, virtual reality, and / or mixed reality applications to provide a computer generated reality, virtual reality, augmented reality, and / or augmented virtual experience to a user.

[0102] Figure 6 FIG. 2 shows a flash-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a flash-assisted gaze tracking system (e.g., Figure 1 and Figure 5 The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and the flash in the current frame. When not in the tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and the flash in the current frame, and if successful, initializes the tracking state to "yes" and continues with the next frame in the tracking state.

[0103] like Figure 6 As shown in , the gaze tracking camera can capture left and right images of the user's left and right eyes. The captured images are then input to the gaze tracking pipeline for processing starting at 610. As indicated by the arrow returning to element 600, the gaze tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each group of captured images can be input to the pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are processed by the pipeline.

[0104] At 610, for the currently captured image, if the tracking status is yes, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as indicated at 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0105] At 640, if advancing from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from previous frames. At 640, if advancing from element 630, the tracking state is initialized based on the pupils and glints detected in the current frame. The processing results at element 640 are checked to verify that the results of tracking or detection can be credible. For example, the results can be checked to determine whether the pupil and a sufficient number of glints for performing gaze estimation are successfully tracked or detected in the current frame. At 650, if the results are not likely to be credible, at element 660, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the results are credible, the method advances to element 670. At 670, the tracking state is set to YES (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0106] Figure 6 It is intended to be used as an example of an eye tracking technology that can be used for a particular implementation. As recognized by one of ordinary skill in the art, according to various embodiments, other eye tracking technologies currently existing or developed in the future can be used in place of or in combination with the flash-assisted eye tracking technology described herein in a computer system 101 for providing an XR experience to a user.

[0107] In some embodiments, the captured portion of the real-world environment 602 is used to provide an XR experience to the user, such as a mixed reality environment in which one or more virtual objects are superimposed on top of a representation of the real-world environment 602 .

[0108] Thus, the description herein describes some embodiments of a three-dimensional environment (e.g., an XR environment) including representations of real-world objects and representations of virtual objects. For example, the three-dimensional environment optionally includes a representation of a table present in a physical environment, which is captured and displayed in the three-dimensional environment (e.g., actively displayed via a camera and display of a computer system or passively displayed via a transparent or translucent display of a computer system). As previously described, the three-dimensional environment is optionally a mixed reality system, wherein the three-dimensional environment is based on a physical environment captured by one or more sensors of a computer system and displayed via a display generation component. As a mixed reality system, the computer system is optionally capable of selectively displaying parts and / or objects of the physical environment so that the corresponding parts and / or objects of the physical environment appear as if they exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system is optionally capable of displaying virtual objects in a three-dimensional environment by placing virtual objects at corresponding positions in the three-dimensional environment that have corresponding positions in the real world to appear as if the virtual objects exist in the real world (e.g., a physical environment). For example, the computer system optionally displays a vase so that the vase appears as if a real vase is placed on top of a table in the physical environment. In some embodiments, a corresponding position in the three-dimensional environment has a corresponding position in the physical environment. Thus, when a computer system is described as displaying a virtual object at a corresponding position relative to a physical object (e.g., such as a position at or near a user's hand or at or near a physical table), the computer system displays the virtual object at a specific position in the three-dimensional environment so that it appears as if the virtual object is at or near the physical object in the physical environment (e.g., the virtual object is displayed at a position in the three-dimensional environment that corresponds to the position in the physical environment where the virtual object would be displayed if it were a real object at that specific position).

[0109] In some embodiments, real-world objects present in the physical environment that are displayed in the three-dimensional environment (e.g., and / or visible via a display generation component) can interact with virtual objects that only exist in the three-dimensional environment. For example, the three-dimensional environment may include a table and a vase placed on top of the table, where the table is a view (or representation) of a physical table in the physical environment and the vase is a virtual object.

[0110] Similarly, the user is optionally able to use one or both hands to interact with virtual objects in the three-dimensional environment as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the computer system optionally capture one or more hands of the user and display representations of the user's hands in the three-dimensional environment (e.g., in a manner similar to displaying real-world objects in the three-dimensional environment described above), or in some embodiments, due to the transparency / translucency of a portion of the user interface being displayed by the display generation component, or due to the projection of the user interface onto a transparent / translucent surface or the projection of the user interface onto the user's eyes or into the field of view of the user's eyes, the user's hands can be seen via the display generation component, via the ability to see the physical environment through the user interface. Therefore, in some embodiments, the user's hands are displayed at corresponding locations in the three-dimensional environment and are treated as if they are objects in the three-dimensional environment, and these objects can interact with virtual objects in the three-dimensional environment as if these virtual objects are physical objects in the physical environment. In some embodiments, the computer system can update the display of the representation of the user's hands in the three-dimensional environment in conjunction with the movement of the user's hands in the physical environment.

[0111] In some of the embodiments described below, the computer system is optionally capable of determining an "effective" distance between a physical object in the physical world and a virtual object in a three-dimensional environment, for example, to determine whether the physical object is directly interacting with the virtual object (e.g., whether the hand is touching, grabbing, holding, etc. a virtual object or is within a threshold distance of the virtual object). For example, a hand that directly interacts with a virtual object optionally includes one or more of the following: a finger of a hand pressing a virtual button, a user's hand grabbing a virtual vase, a user's hand closing together and pinching / holding the user interface of an application, and two fingers that perform any other type of interaction described herein. For example, when determining whether a user is interacting with a virtual object and / or how a user is interacting with a virtual object, the computer system optionally determines the distance between the user's hand and the virtual object. In some embodiments, the computer system determines the distance between the user's hand and the virtual object by determining the distance between the position of the hand in the three-dimensional environment and the position of the virtual object of interest in the three-dimensional environment. For example, the one or more hands of the user are located at a specific location in the physical world, and the computer system optionally captures the one or more hands and displays the one or more hands at a specific corresponding location in the three-dimensional environment (e.g., if the hand is a virtual hand instead of a physical hand, the location where the hand will be displayed in the three-dimensional environment). The location of the hand in the three-dimensional environment is optionally compared with the location of the virtual object of interest in the three-dimensional environment to determine the distance between the one or more hands of the user and the virtual object. In some embodiments, the computer system optionally determines the distance between the physical object and the virtual object by comparing the location in the physical world (e.g., instead of comparing the location in the three-dimensional environment). For example, when determining the distance between the one or more hands of the user and the virtual object, the computer system optionally determines the corresponding position of the virtual object in the physical world (e.g., if the virtual object is a physical object instead of a virtual object, the location where the virtual object will be located in the physical world), and then determines the distance between the corresponding physical location and the one or more hands of the user. In some embodiments, the same technology is optionally used to determine the distance between any physical object and any virtual object. Therefore, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, a computer system optionally executes any of the techniques described above to map the position of the physical object to a three-dimensional environment and / or to map the position of the virtual object to the physical environment.

[0112] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed to, and / or where and what the physical stylus held by the user is pointed to. For example, if the user's gaze is directed to a particular location in the physical environment, the computer system optionally determines the corresponding location in the three-dimensional environment (e.g., the virtual location of the gaze), and if a virtual object is located at the corresponding virtual location, the computer system optionally determines that the user's gaze is directed to the virtual object. Similarly, the computer system is optionally able to determine the direction in which the stylus is pointing in the physical environment based on the orientation of the physical stylus. In some embodiments, based on this determination, the computer system determines a corresponding virtual location in the three-dimensional environment that corresponds to the location in the physical environment that the stylus is pointing to, and optionally determines that the stylus is pointing to the corresponding virtual location in the three-dimensional environment.

[0113] Similarly, the embodiments described herein may refer to the position of a user (e.g., a user of a computer system) in a three-dimensional environment and / or the position of a computer system in a three-dimensional environment. In some embodiments, the user of the computer system is holding, wearing, or otherwise located at or near the computer system. Therefore, in some embodiments, the position of the computer system is used as a proxy for the position of the user. In some embodiments, the position of the computer system and / or the user in the physical environment corresponds to the corresponding position in the three-dimensional environment. For example, the position of the computer system will be a position in the physical environment (and its corresponding position in the three-dimensional environment), and if the user stands at this position, facing the corresponding part of the physical environment visible via the display generation component, the user will see from this position in the physical environment in the same positioning, orientation and / or size (e.g., in an absolute sense and / or relative to each other) of the objects displayed in the three-dimensional environment by the display generation component of the computer system or visible in the three-dimensional environment via the display generation component. Similarly, if the virtual objects displayed in the three-dimensional environment are physical objects in the physical environment (e.g., physical objects placed at the same locations in the physical environment as the virtual objects are located in the three-dimensional environment, and physical objects that have the same size and orientation in the physical environment as they do in the three-dimensional environment), then the position of the computer system and / or user is the position from which the user would see the positions of the virtual objects in the physical environment at the same positions, orientations, and / or sizes (e.g., in an absolute sense and / or relative to each other and real-world objects) as the virtual objects displayed in the three-dimensional environment by the display generation components of the computer system.

[0114] In the present disclosure, various input methods are described with respect to interaction with a computer system. When an input device or input method is used to provide an example, and another input device or input method is used to provide another example, it should be understood that each example is compatible with the input device or input method described with respect to another example and optionally utilizes the input device or input method. Similarly, various output methods are described with respect to interaction with a computer system. When an output device or output method is used to provide an example, and another output device or output method is used to provide another example, it should be understood that each example is compatible with the output device or output method described with respect to another example and optionally utilizes the output device or output method. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment by a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with the method described with respect to another example and optionally utilizes these methods. Therefore, the present disclosure discloses an embodiment as a combination of features of multiple examples, without exhaustively listing all features of the embodiment in the description of each exemplary embodiment.

[0115] User interface and associated processes

[0116] Attention is now turned to embodiments of a user interface ("UI") and associated processes that may be implemented on a computer system (such as a portable multifunction device or a head mounted device) having display generating components, one or more input devices, and (optionally) one or more cameras.

[0117] 7A to 7E An example is shown of how an electronic device according to some embodiments of the present disclosure displays video subtitles for a media item having content at different depths.

[0118] Fig. 7A The three-dimensional environment 730 being displayed by the electronic device 101 and the top view 714 of the three-dimensional environment 730 are shown. Figures 1 to 6 As described above, the electronic device 101 optionally includes a display generation component 120 (eg, a touch screen) and a plurality of image sensors (eg, Figure 3The image sensor 314 may include one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the electronic device 101 can use to capture one or more images of a user or a portion of a user as the user interacts with the electronic device 101. In some embodiments, the user interface shown below may also be implemented on a head-mounted display that includes a display generation component that displays the user interface to the user, as well as sensors that detect the physical environment, movement of the user's hands (e.g., external sensors facing outward from the user), and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0119] In some embodiments, the three-dimensional environment 730 includes one or more representations of physical objects in the physical environment 732 of the electronic device 101 and / or one or more virtual objects (e.g., objects that are not located in the physical environment 732 but are generated and displayed by the electronic device 101). Fig. 7A , the three-dimensional environment 730 includes a representation 710 of a table in a physical environment 732 of the electronic device 101. In some embodiments, the representation 710 of the table is a photorealistic, simplified, or cartoon representation of the table in the physical environment 732, and is displayed in the three-dimensional environment 730 in response to the image sensor 314 of the electronic device 101 capturing one or more images of the table in the physical environment 732. Alternatively, in some embodiments, the representation 710 of the table and / or other portions of the physical environment 732 are passively visible via the electronic device 101 and / or the display generation component 120 via a transparent or semi-transparent display of the electronic device 101 (e.g., the display generation component 120).

[0120] In addition, if Fig. 7AAs shown, the three-dimensional environment 730 also includes a media user interface object 704 (e.g., a virtual object). In some embodiments, the media user interface object 704 is the user interface of a media application currently running on the electronic device 101 (e.g., the user interface of an application that plays audio and / or video items). In some embodiments, the media user interface object 704 presents a representation of a two-dimensional or three-dimensional media item (e.g., a movie, a television program, a live sports game, etc.). When the media user interface object 704 is presenting a representation of a two-dimensional media item, the media user interface object 704 optionally has a length and / or width (e.g., an X and / or Y boundary) within which the representation of the two-dimensional media item is presented (e.g., the two-dimensional media item is presented in a plane at the media user interface object 704). Conversely, if the media user interface object 704 is presenting a representation of a three-dimensional media item, the media user interface object 704 optionally has a length, width, and / or depth (e.g., an X, Y, and / or Z boundary) within which the representation of the three-dimensional media item is presented. For example, as Fig. 7A As shown in the overhead view 714 in FIG. 1 , the electronic device 101 is currently presenting a representation of the media item 705 within the length, width, and depth of the media user interface object 704 (e.g., the X, Y, and Z boundaries of the media user interface object 704)—which is three-dimensional.

[0121] In some embodiments, the representation of media item 705 includes content at different depths in the representation of media item 705. For example, Fig. 7A , the representation of media item 705 is currently presenting content corresponding to a first playback position 720, which includes actor A 706 (e.g., a first portion of the representation of media item 705) at a first position and a first depth in the representation of media item 705 and actor B 708 (e.g., a second portion of the representation of media item 705) at a second position (e.g., different from the first position) and a second depth (e.g., different from the first depth) in the representation of media item 705. In some embodiments, the depth and / or position of one or more portions of the representation of media item 705 are determined by a content creator of media item 705. In some embodiments, the depth of a portion of the representation of media item 705 corresponds to the depth and / or distance of the portion of the representation of media item 705 from the user's viewpoint.

[0122] In addition, if Fig. 7A As shown, the electronic device 101 is displaying the video subtitle user interface element 712. FIG. 11A to FIG. 11C As described in more detail in 10 and method 1200, in some embodiments, the video caption user interface element 712 includes a back panel element and text (e.g., captions or subtitles) corresponding to the spoken and / or non-spoken audio present at the first playback location 720. For example, at Fig. 7A , actor B 708 is currently speaking at first playback position 720, and as a result, the words spoken by actor B 708 are displayed in text form at video caption user interface element 712. In some embodiments, if audio (e.g., speech) is not presented at first playback position 720, electronic device 101 optionally does not display video caption user interface element 712 when the representation of media item 705 is at first playback position 720.

[0123] In some embodiments, the electronic device 101 displays the user interface element 712 at an X position (e.g., a horizontal position relative to the viewpoint of the user 716), a Y position (e.g., a vertical position relative to the viewpoint of the user 716), and / or a Z position (e.g., a distance relative to the viewpoint of the user 716) based on the X, Y, and / or Z position of the portion of the representation of the media item 705 to which the user 716's attention is directed (e.g., the user of the electronic device 101). As described in more detail in method 1000, the attention of the user 716 is optionally determined based on the gaze 740 of the user 716 and / or based on the head positioning of the user 716. For example, in Fig. 7A , the electronic device 101 has optionally detected / determined that the attention of the user 716 is directed toward the actor A 706 in response to detecting via one or more sensors 314 that the gaze 740 of the user 716 is directed toward the actor A 706 and / or in response to detecting via one or more sensors 314 that the head of the user 716 is oriented toward the actor A 706.

[0124] In response to the electronic device 101 detecting / determining that the user 716's attention is directed toward the actor A 706, the electronic device 101 optionally Fig. 7A The user interface element 712 is displayed at the position indicated in Fig. 7A 706, the X, Y, and Z positions of the user interface element 712 in the image corresponding to the X, Y, and Z positions of the actor A 706 (e.g., the user interface element 712 is displayed at an X position offset from (to the right of) the actor A 706, at a Y position offset from (downward from) the actor A 706, and at a Z position of 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm, or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of the actor A 706). It should be noted that although the above embodiment describes an example in which the user interface element 712 is displayed below and to the right of the actor A 706 in response to detecting that the attention of the user 716 is directed toward the actor A 706, it should be understood that the user interface element 712 may also be displayed offset from the actor A 706 in any direction or overlaid on the actor A 706 in response to detecting that the attention of the user 716 is directed toward the actor A 706.

[0125] Also like Fig. 7A , the electronic device 101 is displaying the video caption user interface element 712 at a first distance from the viewpoint of the user 716 (who is currently located at the center of the three-dimensional environment 730 and oriented toward the top / back of the three-dimensional environment 730). In some embodiments, the distance from the viewpoint of the user 716 at which the electronic device 101 displays the video caption user interface element 712 (e.g., the Z position of the video caption user interface element 712) is based on the depth of the portion of the representation of the media item 705 to which the user 716's attention is currently directed. For example, in Fig. 7A , the video caption user interface element 712 is optionally displayed at a first distance from the viewpoint of the user 716 ( Fig. 7A 706) because the electronic device 101 begins displaying the video subtitles user interface element 712 at the same or similar depth as actor A 706 in the representation of the media item 705 (e.g., 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm, or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of actor A 706) in response to the electronic device 101 detecting that the attention of the user 716 is currently directed toward actor A 706. In some embodiments, the electronic device 101 displays the video subtitles user interface element 712 at a specific depth and / or distance from the user's viewpoint using a stereoscopic depth effect.

[0126] In some embodiments, the distance at which the electronic device 101 displays the video caption user interface element 712 from the user's 716 viewpoint is not based on the depth of the portion of the representation of the media item 705 to which the user's attention is directed, but rather based on the depth of the portion of the representation of the media item 705 that is closest to the user's 716 viewpoint. Fig. 7A As shown in the top view 714 in FIG. 1 , at the first playback position 720, actor A 706 is at a depth closer to the viewpoint of the user 716 in the representation of the media item 705 than actor B 708. Therefore, the video caption user interface element 712 is optionally displayed on the Fig. 7A706 is at a distance from the viewpoint of the user 716 indicated in the overhead view 714 in the image because the electronic device 101 begins displaying the video subtitle user interface element 712 at the same or similar depth as actor A 706 in the representation of the media item 705 (e.g., 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of actor A 706) in response to the electronic device 101 determining that actor A 706 is the portion of the representation of the media item 705 closest to the viewpoint of the user 716.

[0127] In some embodiments, as the electronic device 101 plays the representation of the media item 705, the representation of the media item 705 progresses through the content corresponding to different playback positions. Figure 7B , the playback position within the representation of media item 705 has been moved from the first playback position 720 (e.g. Fig. 7A ) advances to the second playback position 724 (as shown in the progress bar in ) in the representation of the media item 705. Figure 7B In response to the playback position in the representation of the media item 705 changing from the first playback position 720 to the second playback position 724, the electronic device 101 has updated the representation of the media item 705 to include content corresponding to the second playback position 724, as shown in FIG. Figure 7B Specifically, when the representation of media item 705 corresponds to second playback position 724, the representation of media item 705 includes a third depth in the representation of media item 705 (e.g., at a depth greater than the depth in the representation of media item 705). Fig. 7A Actor A 706 at a depth further from the viewpoint of user 716 and Fig. 7A Actor B 708 is shown at the same depth in the representation of media item 705 (eg, the depth of actor B 708 does not change between the first playback position 720 and the second playback position 724).

[0128] In addition, if Figure 7B As shown, because actor A 706 is now being displayed at the third depth in the representation of media item 705, actor A 706 is no longer the portion of the representation of media item 705 closest to the viewpoint of user 716, and actor B 708 is now the portion of the representation of media item 705 closest to the viewpoint of user 716. In some embodiments, the distance at which electronic device 101 displays video caption user interface element 712 from the viewpoint of user 716 changes as the distance between the viewpoint of user 716 and the portion of the representation of media item 705 closest to the viewpoint of user 716 changes. For example, in Figure 7B , the electronic device 101 is optionally no longer in Fig. 7AThe video caption user interface element 712 is displayed at a first distance from the viewpoint of the user 716 as described and shown in FIG. 7 because the actor A 706 is no longer the portion of the representation of the media item 705 that is closest to the viewpoint of the user 716 (eg, Fig. 7A 714 in the top view). On the contrary, Figure 7B As shown in the top view 714 in FIG. 1 , the electronic device 101 is now at a second distance from the viewpoint of the user 716 (e.g., at a distance from the user 716). Fig. 7A The video subtitles user interface element 712 is displayed at a greater distance from the viewpoint of the user 716 than at a greater distance from the viewpoint of the user 716 because the video subtitles user interface element 712 is now displayed at the same or similar depth as the actor B 708 in the representation of the media item 705 (e.g., 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of the actor B 708) in response to the electronic device 101 determining that the actor B 708 is now the portion of the representation of the media item 705 that is closest to the viewpoint of the user 716.

[0129] In some embodiments, the electronic device 101 updates / changes the distance of the video caption user interface element 712 from the user's 716 viewpoint not in response to determining that the portion of the representation of the media item 705 closest to the user's 716 viewpoint changes (as described above), but rather in response to the electronic device 101 detecting a change in the user's 716 attention. Figure 7B , the distance of the electronic device 101 displaying the video caption user interface element 712 from the viewpoint of the user 716 is optionally changed from Fig. 7A The first distance indicated in the top view 714 in FIG. 7 is changed to Figure 7B The second distance indicated in the top view 714 in FIG. Fig. 7A 706 (which is farther than the viewpoint of the user 716) because the electronic device 101 detects that the gaze 740 of the user 716 is directed from the actor A 706 (such as Fig. 7A ) is changed to point to actor B 708 (as shown Figure 7B As shown, the video caption user interface element 712 begins to be displayed at the same or similar depth as actor B 708 in the media user interface object 704 (e.g., 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm or 2 mm or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm or 50 cm in front of actor B 708).

[0130] Although Figure 7B The electronic device 101 is comparing Fig. 7A708), the electronic device 101 may display the video captions user interface element 712 farther from the viewpoint of the user 716 (e.g., because when the user 716's attention changes from actor A 706 to actor B 708, actor A 706 and actor B 708 are at different depths in the media user interface object 704), but it should be understood that if actor B 708 is presented at the same depth as actor A 706 when the user 716's attention changes from actor A 706 to actor B 708, then the distance from the viewpoint of the user 716 at which the electronic device 101 is displaying the video captions user interface element 712 will optionally not change. In addition, although Figure 7B shows the coverage with Fig. 7A 705) in the same media item 705, but it should be understood that the electronic device 101 may alternatively respond to the electronic device 101 detecting that the user 716 is shifting his attention from the video caption user interface element 712 to the video caption user interface element 712. Fig. 7A The actor A 706 in is changed to Figure 7B Actor B 708 in 9A to 9E The video caption user interface element 712 is displayed overlaid on (or offset from) actor B 708 in a manner similar to that described in .

[0131] In some embodiments, as the video captions user interface element 712 is displayed closer to or farther from the viewpoint of the user 716, the electronic device 101 displays the video captions user interface element 712 at the same / consistent angular size (e.g., the amount of space that the video captions user interface element 712 occupies in the field of view of the user 716 does not change). Figure 7B is shown as Fig. 7A is farther from the viewpoint of user 716 (e.g. Fig. 7A and Figure 7B ), the angular size of the video caption user interface element 712 is Fig. 7A and Figure 7B The angular size of the video caption user interface element 712 does not change either (eg, because the device 101 appropriately changes the size of the video caption user interface element 712 in the three-dimensional environment so that the angular size of the video caption user interface element 712 does not change).

[0132] Alternatively, in some embodiments, as the video captions user interface element 712 is displayed closer to or farther from the viewpoint of the user 716, the angular size of the video captions user interface element 712 changes (e.g., the angular size of the video captions user interface element 712 optionally increases as the video captions user interface element 712 is displayed closer to the viewpoint of the user 716, and optionally decreases as the video captions user interface element 712 is displayed farther from the viewpoint of the user 716), because the size of the video captions user interface element 712 does not change in a three-dimensional environment, as also described in more detail in method 800. For example, in Figure 7B As an alternative to the electronic device 101 displaying the video caption user interface element 712 at a consistent angular size regardless of the distance from the viewpoint of the user 716 at which the electronic device 101 is displaying the video caption user interface element 712, the electronic device 101 may optionally display the video caption user interface element 712 at a consistent angular size. Fig. 7A The video captions user interface element 712 is displayed at a smaller angular size than the video captions user interface element 712 because the video captions user interface element 712 is displayed at a smaller angular size than the video captions user interface element 712. Figure 7B is shown as Fig. 7A is farther from the viewpoint of user 716.

[0133] In addition, if Figure 7B As shown, the text (e.g., subtitles) included in the video subtitles user interface element 712 is displayed in Fig. 7A and Figure 7B The text in the video caption user interface element 712 is optionally changed between Fig. 7A and Figure 7B , because the spoken and / or non-spoken audio in the representation of the media item 705 changes between the first playback position 720 and the second playback position 724. For example, the text included in the video caption user interface element 712 is optionally changed between the first playback position 720 and the second playback position 724. Fig. 7A and Figure 7B Because the words spoken by actor B 708 at the second playback position 724 in media item 705 are different from those at Fig. 7A The words spoken by actor B 708 at the first playback location 720 are shown to be different.

[0134] In some embodiments, electronic device 101 displays playback controls for a representation of media item 705, such as Figure 7C . In some embodiments, the electronic device 101 begins displaying the playback control user interface 726 in response to detecting that the hand 728 of the user 716 is in a ready state configuration (optionally, while the user's attention is directed to the representation of the media item 705). As previously described, in some embodiments, when the electronic device 101 detects that the hand 728 of the user 716 is in a predetermined hand shape (such as Figure 7CWhen the pre-tap hand shape (and / or pre-pinch hand shape) is shown, the user's 716 hands 728 are optionally in a ready state configuration / pose.

[0135] In some embodiments, playback control user interface 726 includes one or more selectable options for modifying playback of a representation of media item 705. For example, Figure 7C As shown, playback control user interface 726 includes selectable options 726a-726i. In some embodiments, selectable option 726a, when selected, causes electronic device 101 to present a representation of media item 705 in an immersive presentation mode (as will be shown in FIG. FIG. 9B to FIG. 9E 705). In some embodiments, optional option 726b, when selected, causes the electronic device 101 to display a representation of media item 705 in a picture-in-picture presentation mode. In some embodiments, optional option 726c, when selected, causes the electronic device 101 to replace the content presented in the media user interface object 704 with a content playback queue, which includes one or more representations of content items that can be selected to cause the corresponding content item to be played back. In some embodiments, optional option 726d, when selected, causes the electronic device 101 to rewind the representation of media item 705 to a predetermined amount (e.g., 10 seconds, 15 seconds, 20 seconds, 30 seconds, 40 seconds, or 60 seconds). In some embodiments, optional option 726e, when selected, causes the electronic device 101 to pause playback of the representation of media item 705 (e.g., if the representation of media item 705 is currently being played at the media user interface object 704) or play the representation of media item 705 (e.g., if the representation of media item 705 is currently being paused at the media user interface object 704). In some embodiments, selectable option 726f, when selected, causes the electronic device 101 to fast forward the representation of media item 705 by a predetermined amount (e.g., 10 seconds, 15 seconds, 20 seconds, 30 seconds, 40 seconds, or 60 seconds). In some embodiments, selectable option 726g, when selected, causes the electronic device 101 to display a menu associated with the representation of media item 705, the menu including a selectable option for toggling on / off the display of subtitles for the representation of media item 705. In some embodiments, selectable option 726h, when selected, causes the electronic device 101 to modify the playback volume of the representation of media item 705. Finally, as Figure 7C As shown, in some embodiments, the playback control user interface 726 also includes an eraser bar 726i that includes an indication of the current playback position of the representation of the media item 705.

[0136] In some embodiments, when the electronic device 101 is displaying the playback control user interface 726, the electronic device 101 displays the video caption user interface element 712 at a different (e.g., X, Y, and / or Z) position than when the electronic device 101 is not displaying the playback control user interface 726. Figure 7C As shown, when the electronic device 101 does not display the playback control user interface 726 Figure 7B In contrast, while the electronic device 101 is displaying the playback control user interface 726, the electronic device 101 is displaying the video caption user interface element 712 at a higher Y position (eg, a vertical position relative to the user's viewpoint).

[0137] In some embodiments, the electronic device 101 displays the playback control user interface 726 at a height that is the same as or similar to the height at which the video subtitles user interface element 712 was displayed before the electronic device 101 detected the request to display the playback control user interface 726. In some embodiments, the electronic device 101 also displays the playback control user interface 726 at a distance that is closer to the viewpoint of the user 716 than the video subtitles user interface element 712. Therefore, if the electronic device 101 does not begin to display the video subtitles user interface element 712 at a higher Y position in response to the electronic device 101 detecting the request to display the playback control user interface 726, the video subtitles user interface element 712 may be obscured from the viewpoint of the user 716. It should be understood that although the video subtitles user interface element 712 is displayed at a higher Y position in response to the electronic device 101 detecting the request to display the playback control user interface 726, the video subtitles user interface element 712 may be obscured from the viewpoint of the user 716. Figure 7C 712 is not obscured by the playback control user interface 726 from the viewpoint of the user 716, but the electronic device 101 may optionally display the video captions user interface element 712 at a lower Y position, at a different X position, or a combination thereof, which will result in the video captions user interface element 712 not being obscured by the playback control user interface 726 from the viewpoint of the user 716.

[0138] In some embodiments, when the electronic device 101 is displaying the playback control user interface 726, the electronic device 101 stops displaying the video caption user interface element 712 instead of updating the X, Y, and / or Z position of the video caption user interface element 712 as described above. Figure 7C In response to the electronic device 101 detecting a request to display the playback control user interface 726, the electronic device 101 optionally does not redisplay the video subtitle user interface element 712 at a higher Y position, but instead stops displaying the video subtitle user interface element 712 until the playback control user interface 726 is no longer displayed in the three-dimensional environment 730 (e.g., in response to detecting that the hand 728 of the user 716 is no longer in a ready state).

[0139] Additionally, the video subtitles user interface element 712 is Figure 7C Zhongzheng is shown as Figure 7B is farther from the viewpoint of user 716 because Figure 7B Actor B 708 is being presented in the media representation Figure 7C At a depth farther from the viewpoint of the user 716 (e.g. Figure 7B and Figure 7C Since the playback position in the representation of the media item 708 is changed from the second playback position 724 (as shown in the top view 714 in FIG. 1 ), the playback position in the representation of the media item 708 is changed from the second playback position 724 (as shown in FIG. 1 ). Figure 7B ) is changed to the third playback position 729 (as shown Figure 7C ), so the depth of actor B 705 in the representation of the media item is optionally Figure 7B and Figure 7C Change between.

[0140] exist Fig.7D In response to the playback position in the representation of the media item 705 being changed from the third playback position 729 (e.g. Figure 7C ) is changed to the fourth playback position 733 (as shown Fig.7D ), the electronic device 101 is now displaying a representation of the media item 705 at the fourth playback position 733. When the electronic device 101 is presenting the representation of the media item 705 at the fourth playback position 733, the representation of the media item 705 includes the actor B 708 at a fourth depth in the representation of the media item 705 (e.g., at a depth closer to the viewpoint of the user 716 in the representation of the media item 705 than when the representation of the media item 705 was at the third playback position 729) and the actor B 709. Figure 7C Actor A 706 is shown at the same depth in the representation of media item 705 (eg, the depth of actor A 706 does not change between third playback position 729 and fourth playback position 733).

[0141] In addition, Fig.7D , the electronic device 101 has determined that the audio of the representation of the media item 705 corresponds to actor A 706 at the fourth playback location 733 (e.g., because actor A 706 is currently speaking at the fourth playback location 733). In some embodiments, the electronic device 101 displays the video caption user interface element 712 at the X, Y, and / or Z location of the portion of the representation of the media item 705 that corresponds to the current audio of the media item 705. For example, Fig.7D As shown, in response to electronic device 101 determining that the audio of the representation of media item 705 corresponds to actor A 706, electronic device 101 has stopped in Figure 7C The video caption user interface element 712 is displayed at the position indicated in Fig.7D 706 and at the same or similar depth as actor A 706 (e.g., 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of actor A 706). In some embodiments, the electronic device 101 displays the video caption user interface element 712 at a position and / or distance indicated in the figure (e.g., offset from actor A 706 and at a depth that is the same or similar as actor A 706 (e.g., 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of actor A 706). Figure 7C and Fig.7D 708 to Actor A 706 (e.g., for similar reasons as described above, the position of the video caption user interface element 712 is changed between Figure 7C and Fig.7D between changes).

[0142] exist Fig. 7E In the example, the electronic device 101 has detected that the viewpoint of the user 716 has changed from Fig.7D The viewpoint indicated in the top view 714 in FIG. Fig. 7E The viewpoint of user 716 is optionally moved from Fig.7D The viewpoint indicated in moves to Fig. 7E , as the user 716 moves to a corresponding pose and / or position in the physical environment 732. Fig. 7E As shown, in response to the electronic device 101 detecting the movement of the viewpoint of the user 716, the electronic device 101 is now viewing the new viewpoint of the user 716 of the three-dimensional environment 730 ( Fig.7D The three-dimensional environment 730 is displayed from the viewpoint indicated in the top view 714 in FIG. Fig. 7E As shown, in response to the playback position in the representation of the media item 705 changing from the fourth playback position 733 to the fifth playback position 734 , the electronic device 101 is now presenting the representation of the media item 705 at the fifth playback position 734 .

[0143] In some embodiments, as the electronic device 101 is detecting movement of the viewpoint of the user 716 of the three-dimensional environment 730, the electronic device 101 moves the video caption user interface element 712 according to the change (e.g., speed, amount, and / or direction) of the viewpoint of the user 716. For example, when the electronic device 101 is detecting that the viewpoint of the user 716 is moving from Fig.7D Move right to Fig. 7E When and / or from the user's point of view Fig. 7E, the electronic device 101 optionally moves the video caption user interface element 712 from the middle left portion (eg, Figure 7C ) to cover the middle center portion of the representation (as shown Fig.7D shown). Although Fig. 7E The electronic device 101 is shown to move the video captions user interface element 712 to the right in response to the electronic device detecting that the viewpoint of the user 716 moves to the right, but it should be understood that if the viewpoint of the user 716 moves in an upward, downward, or left direction instead, the electronic device 101 will optionally move the video captions user interface element 712 in an upward, downward, or left direction, respectively. In some embodiments, the movement of the user's viewpoint causes the electronic device 101 to move the video captions user interface element 712 in a direction corresponding to the movement of the viewpoint to maintain the video captions user interface element 712 within the user's field of view. In some embodiments, the electronic device 101 moves the video captions user interface element 712 upward to the boundary of the media user interface object 704, but not beyond the boundary.

[0144] FIG. 8A to FIG. 8I 8 is a flowchart illustrating an exemplary method 800 of displaying video subtitles for a media item having content at different depths according to some embodiments. In some embodiments, the method 800 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, a smart phone, a wearable computer, or a head mounted device, the computer system including a display generating component (e.g., Figure 1 , Figure 3 and Figure 4 In some embodiments, method 800 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0145] In some embodiments, method 800 is performed at a computer system (e.g., electronic device 101) in communication with a display generation component (e.g., 120) and one or more input devices (e.g., 314) (e.g., a mobile device (e.g., a tablet, smart phone, media player, or wearable device), or a computer). In some embodiments, the display generation component is a display (optionally a touch screen display) integrated with the computer system, an external display such as a monitor, a projector, a television, or a hardware component (optionally integrated or external) for projecting a user interface or making a user interface visible to one or more users, etc. In some embodiments, the one or more input devices include a computer system or component capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information associated with the user input to the computer system. Examples of input devices include a touch screen, a mouse (e.g., external), a trackpad (optionally integrated or external), a touchpad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the computer system), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, an eye tracking device, and / or a motion sensor (e.g., a hand tracking device, a hand motion sensor), etc. In some embodiments, the computer system communicates with a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, trackpad)). In some embodiments, the hand tracking device is a wearable device, such as a smart glove.

[0146] In some embodiments, the hand tracking device is a handheld input device, such as a remote control or a stylus.

[0147] In some embodiments, when displaying a representation of a media item having content at different depths, such as representation 705, via one or more display generation components (e.g., different segments / portions of the representation of the media item are displayed at different distances from the user's viewpoint). In some embodiments, the representation of the media item is displayed in a three-dimensional environment (e.g., a computer generated reality (CGR) environment), such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment), the computer system overlays (802a) a corresponding user interface element (e.g., 712) on a first portion of the representation of the media item, the user interface element including subtitles for the media item being played by the computer system.

[0148] In some embodiments, the media item being presented by the computer system is an item of video content, such as a movie, a series of episodes or video clips in a series of situational content. In some embodiments, the user interface of the media application presents / displays the representation of the media item. In some embodiments, the computer system displays the corresponding user interface element in response to detecting the user input corresponding to the request for displaying the video subtitle / subtitle of the representation for the media item via one or more input devices (for example, via the selection of optional options or menu items). In some embodiments, if the selection of optional options or menu items is received when the computer system has displayed the corresponding user interface element, the selection causes the computer system to stop the display of the corresponding user interface element (for example, switching off the display of the corresponding user interface element). On the contrary, in some embodiments, if the selection of optional options or menu items is received when the corresponding user interface element is not currently displayed in a three-dimensional environment, the selection causes the computer system to start displaying the corresponding user interface element (for example, switching on the display of the corresponding user interface element). In some embodiments, the computer system displays the corresponding user interface element with a visual appearance according to method 1200. In some embodiments, the position of the corresponding user interface element is based on the user's attention, as described in method 1000. In some embodiments, the corresponding user interface element is covered on the first part of the representation of the media item because the attention of the user of the computer system is currently directed to the first part of the representation of the media item. In some embodiments, the corresponding user interface element has a display offset relative to the part of the representation of the media item to which the user's attention is directed. Therefore, in some embodiments, the part of the representation of the media item on which the computer system covers the corresponding user interface element is optionally different from the part of the representation of the media item to which the user's attention is directed.

[0149] In some embodiments, displaying a corresponding user interface element overlaid on a first portion of the representation of the media item includes displaying the corresponding user interface element at a first distance from a viewpoint of a user of the computer system (802b), such as relative to a first depth in the representation of the media item based on determining that the first portion of the representation of the media item (e.g., a first three-dimensional object) is being presented at a first depth in the representation of the media item. Figure 7B708 and user interface elements 712 in the representation of the media items. In some embodiments, the representation of the media items has a length, width and / or depth (e.g., X, Y and / or Z boundaries) within which the content of the representation of the media items is presented. For example, when the computer system is presenting the representation of the media items at the first playback location, the representation of the media items optionally includes one or more actors / roles, one or more foreground / background objects, and / or background scenery-they are optionally displayed in the boundaries / dimensions of the representation of the media items in a three-dimensional manner. In some embodiments, as the computer system presents the representation of the media items at different playback locations, the computer system updates the representation of the media items to include new, different or additional (e.g., three-dimensional) content in the dimensions / boundaries of the representation of the media items. In some embodiments, the depth at which the content is presented in the representation of the media items is defined by the content creator of the media items. For example, a first portion of the representation of a media item optionally corresponds to a first actor / role, and when the first actor / role is presented at a first depth in the representation of the media item, the first actor / role is a first distance from the user's viewpoint (e.g., 0.3 m, 0.5 m, 0.7 m, 1 m, 1.5 m, 2 m, or 4 m from the user's viewpoint). In some embodiments, based on the depth of the first portion in the representation of the media item, the corresponding user interface element is displayed at a distance from the user's viewpoint. For example, if the first portion of the representation of the media item is displayed at a depth of 0.3 m, 0.5 m, 0.7 m, 1 m, 1.5 m, 2 m, or 4 m from the user's viewpoint in the representation of the media item, the corresponding user interface element is optionally displayed at the same distance from the user's viewpoint or at a distance slightly closer to the user's viewpoint (e.g., 1 cm, 5 cm, 15 cm, 30 cm, 35 cm, 40 cm, 50 cm, 60 cm, or 70 cm closer to the user's viewpoint).

[0150] In some embodiments, based on determining that a first portion of the representation of the media item (e.g., a first three-dimensional object) is being presented at a second depth in the representation of the media item that is different from the first depth, a corresponding user interface element is displayed at a second distance from the user's viewpoint that is different from the first distance (802c), such as relative to Figure 7C708 and user interface element 712 in . For example, the distance at which the corresponding user interface element is displayed from the user's viewpoint changes based on the depth at which the first portion is presented in the representation of the media item. For example, if the computer system is presenting the first portion of the corresponding media item in the representation of the media item at a depth different from the first depth (e.g., presenting at a depth farther or closer to the user's viewpoint than when the first portion was presented at the first depth), the corresponding user interface element is displayed at a different distance from the user's viewpoint. Therefore, if the first portion of the representation of the media item is farther from the user's viewpoint when the computer system is displaying the first portion at the second depth (compared to when the first portion was presented at the first depth), the corresponding user interface element is displayed as farther from the user's viewpoint (e.g., based on the distance of the first portion of the corresponding media item from the user's viewpoint), and if the first portion of the corresponding media item is closer to the user's viewpoint (compared to when the first portion was presented at the first depth), the corresponding user interface element is displayed as closer to the user's viewpoint (e.g., based on the distance of the first portion of the corresponding media item from the user's viewpoint). Changing the distance at which a corresponding user interface element is displayed from a user's viewpoint based on the depth of a portion of a media item from the user's viewpoint provides an efficient way to reduce eyestrain of the user when interacting with the corresponding user interface element and the corresponding media item, thereby enabling the user to use a computer system quickly and efficiently.

[0151] In some embodiments, after presenting the first portion of the representation of the media item at a first depth in the representation of the media item, the computer system displays (804a) the first portion of the representation of the media item at a third depth in the representation of the media item that is different from the first depth, such as actor B 708 from FIG. 7B to FIG. 7C Change depth. In some embodiments, when the computer system is displaying the corresponding user interface element overlaid on the first part of the representation of the media item, the representation of the media item is in the first playback positioning. In some embodiments, due to (for example, in response to) the playback positioning being changed to the second playback positioning from the first playback positioning, the computer system changes from the first part of the representation of the media item presented at the first depth in the representation of the media item to the representation of the media item presented at the second depth. In some embodiments, when the computer system is displaying the first part of the representation of the media item at the third depth, compared with when the computer system is displaying the first part of the representation of the media item at the first depth, the first part of the representation of the media item is farther from the user's viewpoint. In some embodiments, when the computer system is displaying the first part of the representation of the media item at the third depth, compared with when the computer system is displaying the first part of the representation of the media item at the first depth, the first part of the representation of the media item is closer to the user's viewpoint.

[0152] In some embodiments, when displaying the first portion of the representation of the media item at a third depth in the representation of the media item, the computer system displays (804b) the corresponding user interface element at a third distance from the user's viewpoint that is different from the first distance, such as element 712, which is farther from the viewpoint than the first distance. FIG. 7B to FIG. 7C For example, as the first portion of the representation of the media item is displayed at different depths in the representation of the media item (e.g., in response to a change in playback positioning in the representation of the media item), the computer system also updates the distance at which the corresponding user interface element is displayed from the user's viewpoint. Therefore, if the third depth is farther from the user's viewpoint than the first depth, the computer system optionally displays the corresponding user interface element farther from the user's viewpoint than when the computer system is displaying the first portion of the representation of the media item at the first depth. On the contrary, if the third depth is closer to the user's viewpoint than the first depth, the computer system optionally displays the corresponding user interface element closer to the user's viewpoint than when the computer system is displaying the first portion of the representation of the media item at the first depth. Displaying the corresponding user interface element at different distances from the user's viewpoint as the depth of a portion of the representation of the media item changes causes the computer system to automatically display the corresponding user interface at an appropriate distance from the user's viewpoint during playback of the representation of the media item, thereby reducing the user's eye fatigue when interacting with the corresponding user interface element.

[0153] In some embodiments, when the corresponding user interface element is displayed as overlaying a first portion of the representation of the media item, the corresponding user interface element is displayed at a first distance from the user's viewpoint (806a), such as at Figure 7C In some embodiments, after displaying the corresponding user interface element overlaid on a first portion of the representation of the media item at a first distance from the user's viewpoint, the computer system displays (806b) the corresponding user interface element overlaid on a second portion of the representation of the media item that is different from the first portion via one or more display generation components, such as at Fig.7D712 is displayed at its new position in the representation of the media item. For example, in some embodiments, in response to the computer system detecting that the user's attention (e.g., gaze) changes from a first portion of the representation of the media item to a second portion of the representation of the media item, the corresponding user interface element is displayed as overlaying the second portion of the representation of the media item, as described in more detail in method 1000. In some embodiments, in response to detecting that the portion of the representation of the media item closest to the user's viewpoint has changed from the first portion to the second portion (e.g., due to a change in playback positioning in the representation of the media item), the corresponding user interface element is displayed as overlaying the second portion of the representation of the media item. In some embodiments, the corresponding user interface element is displayed as overlaying the second portion of the representation of the media item because the "region of interest" or "target focal plane" defined by the content creator of the media item changes from the first portion to the second portion (e.g., as the playback positioning in the representation of the media item changes).

[0154] In some embodiments, displaying the corresponding user interface element overlying the second portion of the representation of the media item includes, based on determining that the second portion of the representation of the media item is being presented at a third depth in the representation of the media item that is different from the first depth, displaying the corresponding user interface element (806c) at a third distance from the user's viewpoint that is different from the first distance, such as the distance of element 712 from the user's viewpoint being greater than 1. FIG. 7C to FIG. 7DFor example, as the computer system displays the corresponding user interface element overlaying a new portion of the representation of the media item, if the new portion of the representation of the media item is displayed at a different depth in the representation of the media item than the portion of the representation of the media item that the corresponding user interface element previously (e.g., most recently) covered, the distance of the corresponding user interface element from the user's viewpoint changes. Accordingly, if the second portion of the representation of the media item is displayed at a depth closer to the user's viewpoint than the first portion of the representation of the media item (e.g., the portion of the representation of the media item that the corresponding user interface element previously covered), then when the corresponding user interface element is overlaid on the second portion of the representation of the media item, the computer system optionally displays the corresponding user interface element closer to the user's viewpoint. Alternatively, if the second portion of the representation of the media item is displayed at a depth farther from the user's viewpoint than the first portion of the representation of the media item (e.g., the portion of the representation of the media item that the corresponding user interface element most recently covered), then when the corresponding user interface element is overlaid on the second portion of the representation of the media item, the computer system optionally displays the corresponding user interface element farther from the user's viewpoint. In some embodiments, the user of the computer system optionally repositions / moves the corresponding user interface element to cover the third part of the representation of the media item via a pinch and drag gesture. In some embodiments, according to determining that the second part of the representation of the media item is being presented at a fourth depth different from the first depth, the second depth and / or the third depth, the computer system displays the corresponding user interface element at a fourth distance different from the user's viewpoint and the first distance, the second distance and / or the third distance. In some embodiments, according to determining that the second part of the representation of the media item is presented at a second depth different from the first depth, the third depth and / or the fourth depth, the computer system displays the corresponding user interface element at a second distance different from the user's viewpoint and the first distance, the third distance and / or the fourth distance. As the corresponding user interface element covers the different parts of the representation of the media item and displays the corresponding user interface element at different distances from the user's viewpoint, the computer system automatically displays the corresponding user interface at an appropriate distance from the user's viewpoint during the playback of the representation of the media item, thereby reducing the user's eyestrain when interacting with the corresponding user interface element.

[0155] In some embodiments, when one or more criteria are satisfied, including a criterion that is satisfied when the target focus plane corresponds to a first portion of the representation of the media item, the corresponding user interface element is overlaid on the first portion of the representation of the media item, and when the one or more criteria are not satisfied, the corresponding user interface element is not overlaid on the first portion of the representation of the media item (808). For example, Fig. 7AThe target focal plane of the content 705 in the video is optionally a plane defined by the depth of actor A 706. In some embodiments, if the user's gaze points to the first part of the representation of the media item, the target focal plane corresponds to the first part of the representation of the media item. In some embodiments, the target focal plane corresponds to the first part of the representation of the media item, because the content creator defines the first part of the representation as an area of ​​interest (e.g., the area that the content creator is emphasizing, such as a point, plane or volume with a specific location in the media). In some embodiments, the target focal plane changes as the playback positioning of the representation of the media item changes. For example, when the computer system is displaying the representation of the media item at the first playback positioning, the target focal plane optionally corresponds to the first part of the representation of the media item (e.g., because the first part includes the activity that the content creator wishes to emphasize). However, when the computer system starts to display the representation of the media item at the second playback positioning, the target focal plane optionally changes to correspond to the second part of the representation of the media item different from the first part of the representation of the media item (e.g., because the second part now includes the activity that the content creator wishes to emphasize). Displaying a corresponding user interface overlaid on a first portion of a representation of a media item when a target focal plane corresponds to a first portion of a representation of a media item causes a computer system to automatically display corresponding user interface elements at a portion of the representation of the media item designated as a portion of interest during playback of the representation of the media item, thereby reducing eyestrain of a user when interacting with the corresponding user interface elements.

[0156] In some embodiments, displaying a representation of a media item includes simultaneously displaying a first portion of the representation of the media item and a second portion of the representation of the media item that is different from the first portion (810a), such as Fig. 7A In some embodiments, based on determining that the first portion of the representation of the media item is closer to the user's viewpoint than the second portion of the representation of the media item, the computer system overlays (810b) a corresponding user interface element on the first portion of the representation of the media item, such as an overlay on the content 705 that is related to the content 705. Fig. 7A 708 because actor A 706 is closer to the user's viewpoint than actor B 708. For example, the corresponding user interface element is overlaid on the portion of the representation of the media item that is closest to the user's viewpoint. For example, when the second portion of the representation of the media item is displayed at a depth farther from the user's viewpoint in the representation of the media item than the first portion of the representation of the media item, the corresponding user interface element is overlaid on the first portion of the representation of the media item.

[0157] In some embodiments, based on determining that the second portion of the representation of the media item is closer to the user's viewpoint than the first portion of the representation of the media item, the computer system overlays (810c) a corresponding user interface element on the second portion of the representation of the media item, such as an overlay on the content 705. Figure 7B 706 from the user's viewpoint. In some embodiments, the corresponding user interface element is covered on the different parts of the representation of the media item as the computer system plays the representation of the media item, because the part closest to the user's viewpoint of the representation of the media item changes over time. Covering the corresponding user interface element on the part closest to the user's viewpoint of the representation of the media item makes the computer system automatically display the corresponding user interface at an appropriate distance from the user's viewpoint during the playback of the representation of the media item, thereby reducing the eye fatigue of the user when interacting with the corresponding user interface element.

[0158] In some embodiments, while displaying a representation of a media item with a corresponding user interface element overlaid on a first portion of the representation of the media item, the computer system receives (812a) via one or more input devices a first input directed to the representation of the media item, such as Figure 7C In some embodiments, the first input comprises a detected hand 728 in a ready state (e.g., corresponding to a request to display a playback control user interface). In some embodiments, the first input comprises movement of a user's hand. In some embodiments, the first input comprises a hand of the user performing or having performed an air gesture. In some embodiments, the first input comprises a user's hand performing a tap gesture at a location on the display generating component (if the computer system is a touch-sensitive device). In some embodiments, the first input comprises a representation of a user of the computer system directing attention (e.g., gaze) to a media item.

[0159] In some embodiments, in response to receiving the first input (812b), the computer system displays (812c) via one or more display generation components a playback control user interface that includes one or more selectable user interface elements for modifying playback of a representation of the media item, such as Figure 7C726 in the playback control element. In some embodiments, the computer system displays the playback control user interface after (e.g., in response to) determining that the first input satisfies one or more criteria, and does not display the playback control user interface after (e.g., in response to) determining that the first input does not satisfy the one or more criteria. In some embodiments, the one or more criteria are the same or similar to the one or more criteria described below. In some embodiments, one or more optional options for modifying the playback of a representation of a media item include an optional option that causes the computer system to display the media item in an immersive presentation mode when selected, as described in more detail in method 1000. In some embodiments, one or more optional options for modifying the playback of a representation of a media item include an optional option that causes the computer system to display the media item in a picture-in-picture presentation mode when selected. In some embodiments, one or more optional options for modifying the playback of a representation of a media item include an optional option that causes the computer system to present a content playback queue when selected. In some embodiments, one or more selectable options for modifying the playback of a representation of a media item include a selectable option that causes the computer system to rewind the representation of the media item by a predetermined amount (e.g., 5 seconds, 10 seconds, 15 seconds, 30 seconds, or 60 seconds) when selected. In some embodiments, one or more selectable options for modifying the playback of a representation of a media item include a selectable option that causes the computer system to pause / play the representation of the media item when selected. In some embodiments, one or more selectable options for modifying the playback of a representation of a media item include a selectable option that causes the computer system to fast forward the representation of the media item by a predetermined amount (e.g., 5 seconds, 10 seconds, 15 seconds, 30 seconds, or 60 seconds) when selected. In some embodiments, one or more selectable options for modifying the playback of a representation of a media item include a selectable option that causes the computer system to present a subtitle / subtitle option associated with the representation of the media item when selected.

[0160] In some embodiments, the computer system stops (812d) displaying a corresponding user interface element overlying the first portion of the representation of the media item, such as playback control element 726 being displayed overlying the first portion of the representation of the media item. Figure 7C Contents of 705 in Figure 7BIn some embodiments, the playback control user interface of the computer system is displayed in the playback control of the representation of the media items. In some embodiments, the playback control user interface is covered on at least a portion of the representation of the media items covered by the corresponding user interface element before the computer system stops the display of the corresponding user interface element. In response to the request that the computer system detects the playback control of the representation of the media items and stops the display of the corresponding user interface element, the computer system automatically reduces the confusion / hindering of the representation of the media items.

[0161] In some embodiments, the playback control user interface is displayed when one or more criteria are met and is not displayed when the one or more criteria are not met (814a), such as when Figure 7C In some embodiments, after stopping display of the corresponding user interface element and while displaying the playback control user interface (e.g., in response to detecting that the first input satisfies the one or more criteria), such as if in Figure 7C , the computer system determines (814b) that the one or more criteria are not satisfied, such as the hand 728 is no longer in the ready state. In some embodiments, determining that the one or more criteria are satisfied includes determining that the first input includes a movement of a predefined part of the user (e.g., a hand) from a position close to the user's body to a raised position. In some embodiments, determining that the one or more criteria are satisfied includes determining that the first input includes the user's hand in a predefined hand shape / pose (e.g., a ready state pose). In some embodiments, determining that the one or more criteria are satisfied includes determining that the user's attention (e.g., determined based on gaze) is directed to a representation of the media item. In some embodiments, the one or more criteria are not satisfied when one or more input devices of the computer system no longer detect the user's hand that caused (or is associated with) the first input (e.g., because the user's hand is out of range of one or more hand tracking sensors of the computer system). In some embodiments, the one or more criteria are not satisfied when the user's hand that caused (or is associated with) the first input is no longer in a pose that satisfies the one or more criteria. In some embodiments, the one or more criteria are no longer satisfied when the user's attention (e.g., determined based on gaze) is no longer directed toward the representation of the media item.

[0162] In some embodiments, in response to determining that the one or more criteria are not satisfied (814c), the computer system stops (814d) displaying the playback control user interface via one or more display generation components, such as stopping displaying the playback control user interface. Figure 7C . For example, the computer system continues to display the playback control user interface while the one or more criteria remain satisfied, and stops displaying the playback control user interface when the one or more criteria are not satisfied. In some embodiments, the computer system displays (814e) a corresponding user interface element via one or more display generation components in response to detecting that the hand 728 is no longer in the ready state, such as in Figure 7C In some embodiments, the computer system overlays the corresponding user interface element 712 on the representation of the media item. For example, when the computer system stops displaying playback controls, the computer system resumes the display of the corresponding user interface element. In some embodiments, the computer system overlays the corresponding user interface element on the portion of the representation of the media item, which portion is before the computer system starts to display the playback control user interface / the place where the previous computer system was displaying the corresponding user interface element. In some embodiments, when the computer system redisplays the corresponding user interface element, the corresponding user interface element is overlaid on different portions of the representation of the media item (for example, because as the computer system is playing the representation of the media item, the portions of the representation of the media become displayed at different depths) compared to the place where the computer system displayed the corresponding user interface element before displaying the playback control user interface element. Redisplaying the corresponding user interface element after the computer system stops displaying the playback control user interface reduces the number of inputs required to display the corresponding user interface element.

[0163] In some embodiments, in response to receiving the first input and while displaying the playback control user interface (816a), the computer system displays (816b) the corresponding user interface element via the display generation component overlaid on a second portion of the representation of the media item that is different from the first portion, such as when the playback control element 726 is in Figure 7CIn some embodiments, the playback control user interface is displayed by sliding user interface element 712 upwards to different parts of content 705. For example, compared with when the computer system does not display the playback control user interface, when the computer system is displaying the playback control user interface, the corresponding user interface element is covered on the different parts of the representation of the media item. In some embodiments, the playback control user interface is displayed as closer to the user's viewpoint and / or covers (for example, immediately after) the part of the media item covered by the previous corresponding user interface before the computer system displays the playback control user interface. Therefore, in some embodiments, if the computer system continues to display the corresponding user interface element on the first part of the representation of the media item when the playback control user interface is displayed (for example, if the computer system does not reposition the corresponding user interface element from the first part of the representation of the media item to covering on the second part when the playback control user interface is displayed), the corresponding user interface can be hidden from the user's viewpoint. In some embodiments, the playback control user interface and the corresponding user interface element are displayed at the same distance from the user's viewpoint, or at the different distances from the user's viewpoint. In some embodiments, the corresponding user interface element is displayed above or below and / or on the left or right side of the playback control user interface. In some embodiments, the corresponding user interface element and playback control user interface are angled towards the user's viewpoint with the same or different angles.For example, if the playback control user interface is displayed at a lower height than the corresponding user interface element, the computer system optionally makes the playback control user interface angled upward relative to the corresponding user interface element.In some embodiments, in response to the first input, the computer system makes the corresponding user interface element from the first part of the representation of the media item to the second part of the mobile animation.In some embodiments, the orientation of the corresponding user interface element is repositioned / moved and changes as the corresponding user interface element is relocated.For example, if the corresponding user interface element moves vertically upward from the bottom center of the representation of the media item, then when the corresponding user interface element is displayed at the bottom center of the representation of the media item, compared, the corresponding user interface element optionally has less upward tilt (for example, because the corresponding user interface element moves closer to the middle center of the user's visual field).Based on whether the computer system is displaying the playback control user interface and covering the different parts of the representation of the media, the corresponding user interface element is displayed so that the corresponding user interface and playback control can be visible from the user's viewpoint when no additional user input is required, thus reducing the eye fatigue of the user when interacting with the corresponding user interface element.

[0164] In some embodiments, when displaying a corresponding user interface element overlaid on a first portion of a representation of a media item, such as Fig.7D, the computer system detects (818a) a change in the relative position of the representation of the media item relative to the user's viewpoint, such as a change in the user's viewpoint from FIG. 7D to FIG. 7E The change of the relative position of the content 705 of the computer system. For example, the representation of the media item changes from occupying a first position (e.g., the middle center portion) in the field of view from the user's viewpoint to occupying a second position (e.g., the left center portion) in the field of view from the user's viewpoint. In some embodiments, the representation of the media item changes relative to the user's viewpoint due to the movement of the representation of the media item in the three-dimensional environment (e.g., the user's hand selects the representation of the media item using a "pinch gesture", and before releasing the pinch gesture, the user's hand moves in the physical environment of the computer system). In some embodiments, the representation of the media item changes relative to the user's viewpoint because the user's viewpoint changes in the three-dimensional environment. For example, the user's viewpoint changes from being perpendicular to the representation of the media item to being at a different angle / position / orientation relative to the representation of the media item. In some embodiments, as the position of the representation of the media item changes relative to the user's viewpoint, the position of the representation of the media item in the user's field of view also changes. In some embodiments, in response to detecting that a part of the user (e.g., the user's head, torso, hand, etc.) moves (optionally simultaneously with the computer system or display generation component), the user's viewpoint changes.

[0165] In some embodiments, upon detecting a change in the relative position of the representation of the media item relative to the user's viewpoint (818b), the computer system adjusts the media item based on the change in the relative position of the representation of the media item relative to the user's viewpoint, such as the positioning of element 712 above content 705 from FIG. 7D to FIG. 7EThe computer system may also move the corresponding user interface element to cover a different portion of the representation of the media item (compared to the portion of the representation of the media on which the corresponding user interface was displayed before the representation of the media item was moved) from a first portion of the representation of the media item to a second portion of the representation of the media item that is different from the first portion. For example, as the representation of the media item is moving to a new position relative to the user's viewpoint, the computer system may also move the corresponding user interface element to cover a different portion of the representation of the media item (compared to the portion of the representation of the media on which the corresponding user interface was displayed before the representation of the media item was moved). In some embodiments, the second portion of the representation of the media item is selected based on the speed, distance, and / or direction in which the representation of the media item moves relative to the user's viewpoint. In some embodiments, if the representation of the media item moves in a left direction relative to the user's viewpoint, the second portion of the media item (e.g., the portion covered by the corresponding user interface element of the representation of the media item) is to the right of the first portion (e.g., from the user's viewpoint). In some embodiments, if the representation of the media item moves in a downward direction relative to the user's viewpoint, the second portion of the media item (e.g., the portion covered by the corresponding user interface element of the representation of the media item) is above the first portion (e.g., from the user's viewpoint). In some embodiments, the corresponding user interface element moves from overlaying a first portion of the representation of the media item to overlaying a second portion of the representation of the media item because the first portion of the representation of the media item is no longer in the field of view from the user's viewpoint (and the second portion is optionally the portion of the representation of the media item that is closest to the first portion). As the representation of the media item moves relative to the user's viewpoint, moving the corresponding user interface element to overlay a different portion of the representation of the media item causes the computer system to automatically reposition the corresponding user interface element so that it remains visible from the user's viewpoint, thereby reducing eyestrain for the user when interacting with the corresponding user interface element.

[0166] In some embodiments, when one or more criteria are met, including a criterion that the audio of the media item corresponds to the first portion of the representation of the media item, the corresponding user interface element is overlaid on the first portion of the representation of the media item, and when the one or more criteria are not met, it is not overlaid on the first portion of the representation of the media item (820a), such as element 712 overlaying the first portion of the representation of the media item corresponding to the content 705. Figure 7B705, because content 705 currently includes voice from actor B 708. In some embodiments, the corresponding user interface element is overlaid on the first part of the representation of the media item because the spoken and / or non-spoken audio of the media item corresponds to the first part of the representation of the media item (e.g., the actor / role in the first part of the representation of the media item is currently speaking). For example, the computer system is currently playing the audio (e.g., voice) corresponding to the first part of the representation of the media item (e.g., the first role / actor). In some embodiments, if there is no audio (e.g., voice), the computer system does not display the corresponding user interface element. In some embodiments, as the playback positioning of the media item changes, the computer system overlays the corresponding user interface element on the second part of the representation of the media item because the second part now corresponds to the spoken and / or non-spoken audio of the media item (e.g., the actor / role in the second part of the representation of the media item is currently speaking). In some embodiments, if the corresponding user interface element is being displayed at the first part of the representation of the media item, the corresponding user interface element only includes the subtitle text associated with the first part (but not the subtitle / voice text associated with the second part of the representation of the media item). In some embodiments, if the corresponding user interface element is being displayed at the second portion of the representation of the media item, the corresponding user interface element includes only the subtitle text associated with the second portion (and not the subtitle / speech text associated with the first portion of the representation of the media item). Displaying the corresponding user interface element overlaid on the portion of the representation corresponding to the (e.g., spoken and / or non-spoken) audio of the media item causes the computer system to automatically display the corresponding user interface at the current area of ​​interest in the representation of the media item, thereby reducing eyestrain of the user when interacting with the corresponding user interface element.

[0167] In some embodiments, based on determining that the representation of the media item is at a first playback location, the subtitles for the media item include first content corresponding to the audio present at the first playback location (822a), such as Fig. 7A For example, the subtitles displayed in the corresponding user interface element include one or more words spoken by the actor / character at the first playback position, one or more words spoken by the narrator at the first playback position, and / or a text description of the sound event at the first playback position.

[0168] In some embodiments, based on determining that the representation of the media item is at a second playback location different from the first playback location, the subtitles for the media item include second content (822b) that is different from the first content and corresponds to the audio present at the second playback location, such as Figure 7B712 in the corresponding user interface element. For example, as the playback positioning in the representation of media items changes, the computer system updates the subtitles included in the corresponding user interface element. For example, when the representation of media items is in the second playback positioning, the computer system optionally no longer displays the subtitles based on the spoken and / or non-spoken audio at the first playback positioning. On the contrary, the computer system optionally displays the subtitles corresponding to the text description of the sound event at the second playback positioning by one or more words spoken by the actor / role, at the second playback positioning by the narrator, and / or at the second playback positioning in the corresponding user interface element. As the playback positioning of the representation of media items changes, the subtitles shown in the corresponding user interface element are changed so that the computer system automatically presents the relevant information about the media items.

[0169] In some embodiments, while the corresponding user interface element is displayed and while the computer system is playing a representation of the media item, the computer system receives (824a) via one or more input devices an input corresponding to a request to stop displaying the user interface element, such as to stop displaying the media item. Fig. 7A . In some embodiments, the input is received while the computer system is presenting a playback control user interface. In some embodiments, the first input points to a first selectable option that, when selected (e.g., using an air gesture), causes the computer system to toggle off (e.g., stop displaying) subtitles for a representation of the media item (or toggle on subtitles for a representation of the media item if they are not currently displayed). In some embodiments, the first selectable option is displayed in a menu that includes a second selectable option to toggle on display of subtitles for a representation of the media item.

[0170] In some embodiments, in response to receiving the first input, the computer system continues (824b) playback of the representation of the media item without displaying the corresponding user interface element, such as continuing from FIG. 7A to FIG. 7B The content of 705 is played back, but in Figure 7B , element 712 is not displayed in the playback. For example, after the computer system detects user input directed to toggle off display of subtitles for a representation of a media item, the computer system no longer displays the corresponding user interface element during playback of the representation of the media item until the computer system detects a second input corresponding to a request to start displaying subtitles for the representation of the media item. Stopping display of the corresponding user interface element for at least a portion of the remainder of playback of the representation of the media item in response to user input provides an efficient way to manage display of information associated with the representation of the media item.

[0171] It should be understood that the specific order in which the operations in method 800 are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0172] 9A to 9E An example is shown of how an electronic device according to some embodiments of the present disclosure can display video subtitles for a media item based on the attention of a user of the electronic device.

[0173] Fig.9A The three-dimensional environment 920 visible via the display generation component 120 of the electronic device 101 and the top view 914 of the three-dimensional environment 920 are shown. Figures 1 to 6 As described above, the electronic device 101 optionally includes a display generation component 120 (eg, a touch screen) and a plurality of image sensors (eg, Figure 3 The image sensor 314). The image sensor optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the electronic device 101 can use to capture one or more images of the user or a portion of the user when the user interacts with the electronic device 101. In some embodiments, the user interface shown and described below can also be implemented on a head-mounted display, which includes a display generation component that displays the user interface to the user, and sensors that detect the physical environment, movement of the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0174] As mentioned above 7A to 7E As described, in some embodiments, the three-dimensional environment includes one or more representations of physical objects in the physical environment of the electronic device 101 and / or one or more virtual objects (e.g., objects that are not located in the physical environment 922 of the electronic device 101 but are generated and displayed by the electronic device 101). Fig.9A In the example of FIG. 1 , the three-dimensional environment 920 includes a media user interface object 904 (eg, 7A to 7E ) and a representation 910 of a table in a physical environment 922 of the electronic device 101. In some embodiments, one or more portions of the physical environment 922 of the electronic device 101 are passively visible via the electronic device 101 via a transparent or semi-transparent display of the electronic device 101. For example, the representation 910 of the table is optionally not displayed and / or generated by the electronic device 101, but is visible via a transparent or semi-transparent display of the electronic device 101.

[0175] In some embodiments, the electronic device 101 displays the three-dimensional environment 920 from the viewpoint of the user 916 of the electronic device 101. For example, Fig.9A As shown in top view 914 in FIG. 1 , user 916 is currently viewing three-dimensional environment 920 from a position corresponding to the center of three-dimensional environment 920 and in a posture oriented toward the top / back of three-dimensional environment 920. Fig.9A , electronic device 101 is displaying media user interface object 904 and representation 910 of a table because these objects are in the field of view from the current viewpoint of the three-dimensional environment of user 916 (e.g., Fig.9A 914 in the top view). On the contrary, Fig.9A As shown, the electronic device 101 does not display a representation 918 of the sofa (a physical object in the physical environment 922 ) because this object is not currently in the field of view from the current viewpoint of the three-dimensional environment 920 of the user 916 .

[0176] In some embodiments, the viewpoint of the three-dimensional environment 920 of the user 916 corresponds to the position and / or pose of the user 916 in the physical environment 922 (e.g., the operating environment 100) of the electronic device 101. For example, the viewpoint of the three-dimensional environment 920 of the user 916 is optionally Fig.9A 914 in the top view 914 because the user 916 is currently oriented toward a back wall in the physical environment 922 and is located at the center of the physical environment 922 while holding the electronic device 101 (e.g., or wearing the device 101 if the device 101 is a head mounted device). As will be described in more detail below, as the pose of the user 916 in the physical environment 922 changes and / or as the user 916 moves in the physical environment 922, the viewpoint of the user 916's three-dimensional environment 920 optionally changes.

[0177] In some embodiments, the electronic device 101 is configured to present the media item in different presentation modes including an immersive presentation mode and / or a non-immersive presentation mode. Fig.9A , electronic device 101 is currently presenting a representation of media item 905 in a non-immersive presentation mode. In some embodiments, during the non-immersive presentation mode, the content of the media item is presented within the length, width, and / or depth of media user interface object 904. For example, Fig.9A As shown in the overhead view 914 in FIG, when the representation of the media item 905 corresponds to a non-immersive presentation mode, the electronic device 101 is presenting the representation of the media item 905 within the length, width and / or depth (e.g., X, Y and / or Z boundaries) of the media user interface object 904.

[0178] In some embodiments, when the representation of media item 905 corresponds to a non-immersive presentation mode, portions of three-dimensional environment 920 and / or portions of physical environment 922 visible via a semi-transparent or transparent display of electronic device 101 are not obscured / occluded by the representation of media item 905. For example, Fig.9A As shown, during the non-immersive presentation mode, the representation of the media item 905 occupies less than a threshold amount of space in the user's field of view (e.g., less than 75%, 60%, 50%, 40%, 30%, 20%, 10%, or 5% of the space in the user's field of view), and thus, the representation of the media item 905 does not occlude / obstruct one or more other portions of the three-dimensional environment 920 and / or the physical environment 922 from the user's 916 viewpoint.

[0179] As previously referenced 7A to 7E As described with method 800, in some embodiments, the representation of media item 905 includes content (e.g., one or more portions) at different depths within the representation of media item 905. For example, Fig.9A As shown in the overhead view 914 in , the representation of the media item 905 is currently presenting content corresponding to a first playback position 924, which includes actor A 906 at a first position and a first depth in the representation of the media item 905 and a second object 908 (basketball net) at a second position (e.g., different from the first position) and a second depth (e.g., different from the first depth) in the representation of the media item 905.

[0180] In addition, Fig.9A In the embodiment, the electronic device 101 is also presenting the user interface element 912. In some embodiments, the user interface element 912 has a 7A to 7E In some embodiments, the user interface element 912 is associated with one or more features of the video caption user interface element 712 described in 7A to 7E 712 described in the foregoing description and / or includes additional or alternative information than video captions user interface element 712. For example, if Fig.9A If media item 905 in corresponds to a (e.g., live) sporting event, then Fig.9A The user interface element 912 in will optionally include information about the sporting event, such as the current score of the sporting event, the participating teams of the sporting event, and / or the remaining time of the sporting event. Fig.9AIf media item 905 in the video is a movie or television show, user interface element 912 will optionally include information related to the current playback position in media item 905, such as the name and / or biography of the actor / character at the current playback position in media item 905; information about the song playing at the current playback position in media item 905; and / or comments from the director of media item 905 regarding the current playback position in media item 905.

[0181] In some embodiments, when the representation of media item 905 corresponds to a non-immersive presentation, electronic device 101 displays user interface element 912 at an X position (e.g., a horizontal position relative to the viewpoint of user 916), Y position (e.g., a vertical position relative to the viewpoint of user 916), and / or Z position (e.g., a distance from the viewpoint of user 916) corresponding to the X, Y, and / or Z position of the portion of the representation of media item 905 to which user 916's attention is currently directed (e.g., during a non-immersive presentation mode, to 7A to 7E 905 and / or in a similar manner as described in the immersive presentation mode of the representation of the referenced media item 905). In some embodiments, the attention of the user 916 is determined based on the gaze 926 of the user 916 and / or based on the head positioning of the user 916. For example, in response to detecting via one or more sensors 314 that the gaze 926 of the user 916 is directed toward the upper right portion of the representation of the media item 905 and / or in response to detecting via one or more sensors 314 that the head of the user 916 is oriented toward the upper right portion of the representation of the media item 905, the electronic device 101 optionally detects / determines that the user 916 is focused on the upper right portion of the representation of the media item 905. Fig.9A The attention of user 916 in FIG. 9 is directed to the upper right portion of the representation of media item 905 .

[0182] Alternatively, in some embodiments, when the representation of media item 905 corresponds to a non-immersive presentation mode, electronic device 101 optionally displays user interface element 912 at a predetermined X position (e.g., horizontal position relative to the viewpoint of user 916), Y position (e.g., vertical position relative to the viewpoint of user 916), and / or Z position (e.g., distance from the viewpoint of user 916) that is not based on the attention of user 916. For example, Fig.9AAs shown, even if the electronic device 101 has detected that the attention of the user 916 is directed to the upper right portion of the representation of the media item 905, the electronic device 101 does not display the user interface element 912 at the position corresponding to the upper right portion of the representation of the media item 905, but instead displays the user interface element 912 overlying the bottom center portion of the representation of the media item 905 (for example, because even when the attention of the user 916 is directed elsewhere, the electronic device 101 displays the user interface element 912 overlying the predetermined bottom center portion of the representation of the media item 905).

[0183] exist Fig. 9B , the electronic device 101 is now presenting a representation of the media item 905 in an immersive presentation mode and at a second playback location 930 (e.g., with Fig.9A In some embodiments, in response to the electronic device 101 detecting the presence of a signal that is different from the first playback location 924 shown in FIG. Figure 7C , electronic device 101 begins presenting a representation of media item 905 in an immersive presentation mode. In some embodiments, the electronic device 101 begins presenting a representation of media item 905 in an immersive presentation mode, similar to the selection of selectable option 726a described in FIG. Fig.9A ), the representation of media item 905 occupies a greater amount of space in the field of view of user 916 during immersive presentation mode. Fig. 9B As shown in the overhead view 914 in FIG. 1 , the representation of the media item 905 has changed in size from Fig.9A Expands to an area spanning 180 degrees around the viewpoint of user 916. Fig. 9B The representation of media item 905 is shown as occupying an area of ​​180 degrees around the viewpoint of user 916, but one of ordinary skill in the art should understand that the representation of media item 905 during immersive presentation mode may also occupy an area of ​​45 degrees, 90 degrees, 125 degrees, 250 degrees, or 360 degrees around the viewpoint of user 916 without departing from the scope of the present disclosure.

[0184] In some embodiments, during the immersive presentation mode, the electronic device 101 displays the user interface element 912 at an X position (horizontal position) relative to the viewpoint of the user 916 and at a fixed Y position (vertical position) relative to the viewpoint of the user 916 and / or at a Z position (distance) relative to the viewpoint of the user 916, the X position being based on the portion of the representation of the media item 905 to which the user's attention is directed. For example, in an embodiment where the electronic device 101 is displaying the user interface element 912 at a fixed Y position and / or Z position relative to the viewpoint of the user 916, the electronic device 101, in response to detecting that the attention of the user 916 is directed toward the actor A 906, optionally displays the user interface element 912 at position 912 b (at a distance from the viewpoint of the actor A 906). Fig. 9B916 ), because the Y and Z positioning of user interface element 912 at position 912 b corresponds to the vertical position and distance relative to the viewpoint of user 916 that is predetermined by electronic device 101, and because the X positioning of user interface element 912 at position 912 b corresponds to the X positioning of actor A 906 toward which gaze 926 is directed (e.g., user interface element 912 has the same (or similar) X positioning as actor A 906—the portion of the representation of media item 905 toward which user 916's attention is directed).

[0185] The above example describes an embodiment in which the Y positioning of user interface element 912 has been fixed so that user interface element 912 is displayed as overlaying the bottom of the representation of media item 905. However, it should be noted that the Y positioning of user interface element 912 can also be fixed so that user interface element 912 is displayed in the middle or top portion of the representation of media item 905. It should also be noted that in some embodiments, the Z positioning of user interface element 912 is not predetermined (fixed) as described above. Instead, the Z positioning of user interface element 912 can also correspond to the Z positioning of the portion of the representation of media item 905 to which the user's attention is directed (e.g., actor A 906). In addition, although the above embodiment describes an example in which electronic device 101 displays user interface element 912 at a predetermined (fixed) Y positioning and an X positioning based on the attention of user 916, it is also envisioned that the X positioning of user interface element 912 is predetermined / fixed and the Y positioning of user interface element 912 is based on the attention of user 916.

[0186] In embodiments where the X-position of user interface element 912 is based on the attention of user 916 and the Y-position and / or Z-position of user interface element 912 has been predetermined by electronic device 101, the X-position of user interface element 912 optionally changes as the attention of user 916 changes. Fig. 9C In the example, the electronic device 101 has detected that the user 916's attention has shifted from Fig. 9B The direction shown is changed to actor A 906 as shown in FIG. Fig. 9C The pointer is shown pointing to the second object 908, and the playback position is now playback position 931. In response to detecting a change in the attention of the user 916, the electronic device 101 optionally stops displaying the user interface element 912 at position 912b (in Fig. 9B ) and starts at Fig. 9C, which is to the left of position 912b from the viewpoint of user 916. In some embodiments, electronic device 101 moves / slides user interface element 912 from position 912b to position 912d without ceasing display of user interface element 912—in some embodiments, electronic device 101 stops displaying user interface element 912 (e.g., when at position 912b), and then redisplays user interface element 912 at position 912d. Electronic device 101 optionally begins displaying user interface element 912 at position 912d in response to detecting that user 916's attention is directed toward second object 908 because electronic device 101 updates the X position of user interface element 912 to correspond to second object 908 (e.g., to have the same X position as second object 908 in the representation of media item 905). Additionally, as Fig. 9B and Fig. 9C As shown, the Y-position and Z-position of user interface element 912 do not change as electronic device 101 stops displaying user interface element 912 at position 912b and begins displaying user interface element 912 at position 912d because, as described above, in some embodiments, the Y-position and Z-position of user interface element 912 are predetermined / fixed and therefore do not change even as the attention of user 916 changes.

[0187] As an alternative to the above-described embodiments in which user interface element 912 is displayed at a fixed position along at least one respective axis (e.g., X, Y, and / or Z axis), in some embodiments, user interface element 912 is not displayed at a fixed position along two or more axes (e.g., any axis) relative to the viewpoint of user 916. For example, in embodiments in which electronic device 101 does not display user interface element 912 at a fixed position along any respective axis, electronic device 101 optionally displays user interface element 912 at a fixed position along any respective axis in response to electronic device 101 detecting that user 916's attention is directed toward actor A 906. Fig. 9B912a (rather than displaying user interface element 912 at position 912b as previously described). Electronic device 101 optionally displays user interface element 912 at position 912a in response to detecting that user 916's attention is directed toward actor A 906 because the X-position, Y-position, and Z-position of user interface element 912 at position 912a correspond to actor A 906 (e.g., user interface element 912 is displayed at the same X-position as actor A 906, at a Y-position that is offset (below) actor A 906, and at a Z-position that is 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm, or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of actor A 906). Although the above-described embodiments describe examples in which user interface element 912 is displayed below actor A 906 (the portion of the representation of media item 905 to which the user's attention is directed), it should be understood that when user interface element 912 is not displayed in a fixed position along a corresponding axis, user interface element 912 may be displayed offset from actor A 906 in any direction or overlaid on actor A 906 (e.g., in response to detecting that user 916's attention is directed toward actor A 906).

[0188] In embodiments where electronic device 101 does not display user interface element 912 at a fixed position along two or more axes (e.g., any axis), the X, Y, and / or Z position of the user interface optionally changes as the attention of user 916 changes. Fig. 9C In FIG. 1 , electronic device 101 has detected that user 916's attention has changed from being directed at actor A 906 to being directed at second object 908 (as described above). In response, electronic device 101 optionally stops user interface element 912 from being displayed. Fig. 9B The display at position 912a shown in FIG. 1 and starting at Fig. 9C912c shown in the figure (e.g., either sliding / moving user interface element 912, or ceasing to display user interface element 912 and subsequently redisplaying it). The electronic device 101 optionally begins displaying the user interface element 912 at position 912c in response to detecting that the user 916's attention has changed to the second object 908 because the electronic device 101 updates the X, Y, and Z positions of the user interface element 912 to correspond to the second object 908 (e.g., updating the user interface element 912 to be displayed at the same X position as the second object 908, at a Y position that is offset (e.g., below) from the second object 908, and at a Z position that is 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm, or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of the second object 908). The position 912c of the user interface element 912 is optionally aligned with the X, Y, and Z positions of the second object 908 on two or more axes. Fig. 9B Although the above embodiments describe examples in which user interface element 912 is displayed below second object 908 (the portion of the representation of media item 905 to which user 916's attention is directed), it should be understood that when user interface element 912 is not displayed at a fixed position along the corresponding axis, user interface element 912 may be displayed offset from or overlaid on second object 908 in any direction (e.g., in response to detecting that user 916's attention is directed toward actor A 906).

[0189] In some embodiments, as the user's viewpoint of three-dimensional environment 920 changes, electronic device 101 optionally displays different portions of the representation of media item 905. For example, in Fig. 9C and Fig.9D In the 3D environment 920, the user's viewpoint has changed from Fig. 9C The viewpoint shown in the top view 914 moves to Fig.9D In some embodiments, the user 916's 3D environment 920 is moved to a point in time where the user's 916 viewpoint is located. Fig.9D , as the user 916 moves from the physical environment 922 Fig. 9C The posture indicated in the top view 914 of Fig.9D The posture indicated in the top view 914 of .

[0190] like Fig.9D As shown, in response to electronic device 101 detecting movement of user 916's viewpoint, electronic device 101 is now displaying the portion of the representation of media item 905 that is in the field of view from user 916's current viewpoint, and the playback position is playback position 934. Specifically, Fig.9D , the electronic device 101 is now presenting the portion of the representation of the media item that corresponds to the third object 932 because this object is now in the field of view from the viewpoint of the user 916 (eg, Fig.9D 914 in the top view), and the portion of the representation of media item 905 corresponding to actor A 906 is no longer presented because this object is no longer in the field of view from the user's 916 point of view.

[0191] In addition, if Fig.9D As shown, for reasons similar to those described above, electronic device 101 is displaying user interface element 912 at position 912e (e.g., two or more non-fixed axes) or 912f (e.g., two fixed axes). In some embodiments, user interface element 912 exhibits inert follow-up behavior, also as described above. For example, Fig.9D In some embodiments, in response to detecting that the user's attention is directed to the third object 932, the electronic device 101 optionally does not immediately display the user interface element 912 at the location 912e or 912f. Instead, the electronic device 101 optionally delays displaying the user interface element 912 at the location 912e or 912f until the user's 916 attention has been directed to the third object 932 for at least a threshold amount of time (e.g., 0.5 seconds, 1 second, 3 seconds, 7 seconds, or 10 seconds) and / or until other criteria described in the method 1000 are met. In some embodiments, once the user's attention is directed to the third object 932, the electronic device 101 begins to slide / move the user interface element 912 to the location 912e or 912f, but the sliding / moving takes a certain amount of time (e.g., 0.5 seconds, 1 second, 3 seconds, 7 seconds, or 10 seconds) to complete.

[0192] exist Fig.9E In the example, the electronic device 101 detects again that the viewpoint of the user 916 changes from Fig.9D The viewpoint indicated in the top view in the Fig.9E The viewpoint indicated in the top view in FIG. 1 and the playback position is playback position 936. Fig. 9C and Fig.9D For similar reasons as described in , the viewpoint of user 916 is optionally changed from Fig.9D The viewpoint shown in the figure moves to Fig.9E Thus, electronic device 101 is now presenting portions of three-dimensional environment 920 (e.g., outside of representation of media item 905) and portions of representation of media item 905 that are in the field of view of the current viewpoint of three-dimensional environment 920 of user 916 (e.g., outside of representation of media item 905). Fig.9E It should be appreciated that if the representation of media item 905 occupies a 360 degree area around the user's viewpoint of three-dimensional environment 920 rather than Fig.9EThe 180-degree area shown, when the user's viewpoint moves to Fig.9E When the viewpoint is indicated in , portions of three-dimensional environment 920 outside of the representation of media item 905 will optionally not be displayed (e.g., because the representation of media item 905 will obscure those portions).

[0193] In some embodiments, when the electronic device 101 is presenting a representation of the media item 905 in an immersive presentation mode, the representation of the physical environment 922 in the three-dimensional environment 920 is visually de-emphasized. Fig.9E Because the electronic device 101 is presenting a representation of the media item 905 in an immersive presentation mode (as described above), the representations 918 of the sofa and wall in the physical environment 922 are optionally less transparent, more transparent, displayed with lower contrast, and / or displayed with fewer colors during the immersive presentation mode than they would be if the representations of the media item 905 were in a non-immersive presentation mode.

[0194] In some embodiments, when the user's 916 attention becomes directed to an area outside of the representation of the media item 905, the electronic device 101 continues to display the user interface element 912 within the X, Y, and / or Z boundaries of the representation of the media item 905. Fig.9E , electronic device 101 is currently detecting that user 916's attention is directed toward a back wall of three-dimensional environment 920 (e.g., an area of ​​three-dimensional environment 920 that does not correspond to a representation of media item 905 and is outside of the representation of media item 905). Fig.9E As shown, in response to electronic device 101 detecting that user 916's attention is directed to an area outside the representation of media item 905, electronic device 101 still displays user interface element 912 within the boundaries of the representation of media item 905. In some embodiments, user interface element 912 is positioned close to (or as close as possible) to the location of the user's attention outside the representation of media item 905, while user interface element 912 does not extend outside the representation of media item 905 and / or while remaining farther from the outer boundary of the representation of media item 905 than a threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm).

[0195] FIG. 10A to FIG. 10G 1 is a flowchart illustrating a method 1000 for displaying video subtitles for a media item based on the attention of a user of an electronic device according to some embodiments. In some embodiments, the method 1000 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, a smart phone, a wearable computer, or a head mounted device, the computer system including a display generating component (e.g., Figure 1 , Figure 3 and Figure 4 In some embodiments, method 1000 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0196] In some embodiments, method 1000 is performed at a computer system (e.g., electronic device 101) in communication with a display generation (e.g., 120) and one or more input devices (e.g., 314). In some embodiments, the computer system is the same or similar to the computer system described above with reference to method 800. In some embodiments, the one or more input devices are the same or similar to the one or more input devices described above with reference to method 800. In some embodiments, the display generation component is the same or similar to the display generation component described above with reference to method 800.

[0197] In some embodiments, while displaying a representation of a media item (e.g., similar to the representation of a media item described in methods 800 and / or 1200) via a display generation component and while the attention of a user of the computer system is directed toward a first portion of the representation of the media item, the computer system displays (1002a) via the display generation component a user interface element (e.g., such as the user interface element described in methods 800 and / or 1200), such as element 912, that includes information associated with the media item, wherein the user interface element is displayed at a first position relative to the representation of the media item that corresponds to the first portion of the representation of the media item, such as when the user's gaze 926 is directed toward actor A 906. Fig. 9B912 in the position of the element. In some embodiments, the representation of the media item is displayed in a three-dimensional environment. In some embodiments, when the gaze of the user of the computer system points to the first part of the corresponding media item, it is determined that the attention of the user of the computer system points to the first part of the representation of the media item. In some embodiments, the eye tracking device of the one or more input devices detects that the user's gaze points to the first part of the representation of the media item. In some embodiments, the head tracking device of the one or more input devices detects that the user's head points to the first part of the representation of the media item. In some embodiments, the attention of the user of the computer system corresponds to the position of the user interface element relative to the representation of the media item. For example, if the attention of the user of the computer system (e.g., the gaze of the user of the computer system) points to the lower right part of the representation of the media item, the computer system optionally displays the user interface element at a position corresponding to the lower right part of the representation of the media item. In some embodiments, when the user's attention points to the first part of the representation of the media item, the user's attention points to the X, Y and / or Z position corresponding to the first part of the representation of the media item. In some embodiments, when the user's attention is directed to the first portion of the representation of the media item, the user interface element is displayed at an X, Y, and / or Z position that is offset from the X position of the first portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm), from the Y position of the first portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm), and / or from the Z position of the first portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm). In some embodiments, the Z position of the user interface element is based on the distance of the first portion of the corresponding media from the user's viewpoint, as described in method 800. In some embodiments, the user's attention limits the X position of the user interface element (e.g., the horizontal position relative to the user's viewpoint), but does not limit the Y position (e.g., the vertical position relative to the user's viewpoint) and / or Z position (e.g., the distance relative to the user's viewpoint) of the user interface element, as will be described in more detail later. Thus, in some embodiments, the Y and / or Z position of a user interface element is optionally fixed (e.g., held constant) by the computer system, but the X position of the user interface element changes based on where the user's attention is directed. In some embodiments, the user interface element is displayed in a pose / orientation that is perpendicular to the user's viewpoint of the three-dimensional environment.

[0198] In some embodiments, while displaying a representation of a media item, the computer system detects (1002b) via one or more input devices that the attention of a user of the computer system has changed from being directed at a first portion of the representation of the media item to being directed at a second portion of the representation of the media item that is different from the first portion, such as by gazing 926 at a second portion of the representation of the media item. Fig. 9C 908 in the representation of the media item. For example, the computer system has detected that the user's gaze has changed from being directed to the lower right portion of the representation of the media item to being directed to the upper left portion of the representation of the media item. In some embodiments, when the user's attention (e.g., the user's gaze) changes from being directed to a first portion of the representation of the media item to being directed to a second portion of the representation of the media item, the X, Y, and / or Z positioning of the user's attention changes compared to when the user's attention was directed to the first portion of the corresponding media item.

[0199] In some embodiments, after (e.g., in response to) detecting that the attention of a user of the computer system has changed from being directed at a first portion of the representation of the media item to being directed at a second portion of the representation of the media item, the computer system displays (1002c) a user interface element at a second position relative to the representation of the media item that corresponds to the second portion of the media item, such as Fig. 9C912 in the updated position of the element 912. For example, if the user's attention has changed from the lower right portion of the representation pointing to the media item to the upper left portion of the representation pointing to the media item, the user interface element is displayed at the position of the upper left portion of the representation relative to the media item (rather than at the position of the lower right portion of the representation relative to the media item). Therefore, in some embodiments, if the second part of the representation of the media item is on the right side of the first part of the representation of the media item, the user interface element is optionally displayed at a position more to the right than when the user's attention is directed to the first part of the representation of the media item. Similarly, in some embodiments, if the second part of the representation of the media item is above the first part of the representation of the media item, the user interface element is displayed at a position higher (e.g., in the Y direction) than when the user's attention is directed to the first part of the representation of the media item. Similarly, in some embodiments, if the second part of the representation of the media item is farther from the user's viewpoint than the first part of the representation of the media item (e.g., in the Z direction), the user interface element is displayed as farther from the user's viewpoint than when the user's attention is directed to the first part of the representation of the media item. In some embodiments, when the user's attention is directed to the second portion of the representation of the media item, the user interface element is displayed at an X, Y, and / or Z position that is offset from the X position of the second portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm), from the Y position of the second portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm), and / or from the Z position of the second portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm). In some embodiments, when the position of the user interface element changes from a position corresponding to the first portion of the representation of the media item to a position corresponding to the second portion of the representation of the media item, the Y and / or Z positions of the user interface element do not change, but only the X position of the user interface element changes. In some embodiments, the updated X position of the user interface element is based on the X position to which the user's attention is directed. In some embodiments, the user's attention specifies an "active object" in the corresponding media item, and the computer system displays the user interface element based on the position of the "active object." Thus, in some embodiments, if the user's attention changes to a different portion of the "active object," the computer system optionally does not update the position of the user interface element until the user's attention changes to a different object in the representation of the media item.

[0200] Changing the position of user interface elements as the user's attention becomes directed to different parts of the representation of the media item reduces eyestrain on the user when interacting with the corresponding user interface elements and the representation of the media item, and makes it easier for the user to view user interface elements without missing events in the representation of the media item, thereby enabling the user to use the computer system quickly and efficiently.

[0201] In some embodiments, detecting the attention of a user of the computer system includes (1004a) determining (1004b) that the user's attention is directed to the first portion of the representation of the media item based on determining that the user's gaze (e.g., 926) is directed to the first portion of the representation of the media item. In some embodiments, the user's attention is directed to the first portion of the representation of the media item when the user's gaze has been directed to the first portion of the representation of the media item for more than a predetermined threshold amount of time (e.g., 1 second, 2 seconds, 3 seconds, 5 seconds, 10 seconds, 15 seconds, 30 seconds, or 60 seconds). In some embodiments, if the user's gaze has not been directed to the first portion of the representation of the media item for more than the predetermined threshold amount of time, the user's attention is not directed to the first portion of the representation of the media item. In some embodiments, the user's attention is directed to the first portion of the representation of the media item when the user is looking at the first portion of the representation of the media item (e.g., in response to an eye tracking device via one or more input devices detecting that the position of the user's gaze corresponds to the first portion of the representation of the media item). For example, if the first portion of the representation of the media item corresponds to the lower right portion of the representation of the media item, then when the user's gaze is directed to the lower right portion of the representation of the media item (e.g., for more than the predetermined threshold amount of time described above), the user's attention is optionally directed to the lower right portion of the representation of the media item.

[0202] In some embodiments, detecting the attention of a user of the computer system includes (1004a) determining (1004c) that the user's attention is directed to the second portion of the representation of the media item based on determining that the user's gaze (e.g., 926) is directed to the second portion of the representation of the media item. In some embodiments, the user's attention is directed to the second portion of the representation of the media item when the user's gaze has been directed to the second portion of the representation of the media item for more than a predetermined threshold amount of time (e.g., 1, 2, 3, 5, 10, 15, 30, or 60). In some embodiments, if the user's gaze has not been directed to the second portion of the representation of the media item for more than the predetermined threshold amount of time, the user's attention is not directed to the second portion of the representation of the media item. In some embodiments, the user's attention is directed to the second portion of the representation of the media item when the user is looking at the second portion of the representation of the media item (e.g., in response to an eye tracking device via one or more input devices detecting that the position of the user's gaze corresponds to the second portion of the representation of the media item). For example, if the second portion of the representation of the media item corresponds to the upper left portion of the representation of the media item, then when the user's gaze is directed to the upper left portion of the representation of the media item (e.g., for more than the predetermined threshold amount of time described above), the user's attention is optionally directed to the upper left portion of the representation of the media item.

[0203] Using gaze to determine the portion of the representation of the media item to which the user's attention is directed is an efficient way to determine the user's attention.

[0204] In some embodiments, detecting the attention of a user of the computer system includes (1006a) determining (1006b) that the user's attention is directed to the first portion of the representation of the media item based on determining that a (e.g., predefined) portion of the user is directed to the first portion of the representation of the media item. In some embodiments, when a predefined portion of the user (e.g., head, torso, etc.) becomes oriented toward the first portion of the representation of the media item, the user's attention is determined to be directed to the first portion of the representation of the media item. In some embodiments, when a predefined portion of the user (e.g., head) is not oriented toward the first portion of the representation of the media item, the user's attention is not directed to the first portion of the representation of the media item. In some embodiments, the one or more input devices include a head tracking device, and the head tracking device detects the current orientation of the user's head. In some embodiments, when the user's head has been oriented toward the first portion of the representation of the media item for more than a predetermined threshold amount of time (e.g., 1 second, 2 seconds, 3 seconds, 5 seconds, 10 seconds, 15 seconds, 30 seconds, or 60 seconds), the user's attention is directed to the first portion of the representation of the media item. For example, if the first portion of the representation of the media item corresponds to the lower right portion of the representation of the media item, then when the user's head has been oriented toward the lower right portion of the representation of the media item for more than the predetermined threshold amount of time described above, the user's attention is optionally directed to the lower right portion of the representation of the media item.

[0205] In some embodiments, detecting the attention of a user of the computer system includes (1006a) determining (1006c) that the user's attention is directed to the second portion of the representation of the media item based on determining that the portion of the user is directed to the second portion of the representation of the media item. In some embodiments, when a predefined portion of the user (e.g., head) becomes oriented toward the second portion of the representation of the media item, the user's attention is determined to be directed to the second portion of the representation of the media item. In some embodiments, if the predefined portion of the user (e.g., head) is not oriented toward the second portion of the representation of the media item, the user's attention is not directed to the second portion of the representation of the media item. In some embodiments, when the user's head has been oriented toward the second portion of the representation of the media item for more than a predetermined threshold amount of time (e.g., 1 second, 2 seconds, 3 seconds, 5 seconds, 10 seconds, 15 seconds, 30 seconds, or 60 seconds), the user's attention is directed to the second portion of the representation of the media item. For example, if the second portion of the representation of the media item corresponds to the upper left portion of the representation of the media item, then when the user's head has been oriented toward the upper left portion of the representation of the media item for more than the above-mentioned predetermined threshold amount of time, the user's attention is optionally directed to the upper left portion of the representation of the media item.

[0206] Using a predefined part of the user (eg, the head) to determine the part of the representation of the media item to which the user's attention is directed is an efficient way of determining the user's attention.

[0207] In some embodiments, in response to detecting that the attention of a user of the computer system has changed from being directed at a first portion of a representation of a media item to being directed at a second portion of the representation of the media item (1008a), based on determining that the user's attention satisfies one or more criteria after being changed to being directed at the second portion of the representation of the media item (e.g., in some embodiments, the one or more criteria are satisfied if the user's attention has been directed at the second portion of the representation of the media item for at least a threshold amount of time (e.g., 0.5 seconds, 1 second, 3 seconds, 7 seconds, 10 seconds, 20 seconds, or 30 seconds) after the user's attention has been changed to be directed at the second portion of the representation of the media item). In some embodiments, the one or more criteria are satisfied if the user's attention has moved less than a threshold amount (e.g., less than 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 100 cm, 300 cm, or 1000 cm) after the user's attention (e.g., gaze) has been changed to be directed at the second portion of the representation of the media item, the computer system displays (1008b) a user interface element at a second position relative to the representation of the media item that corresponds to the second portion of the representation of the media item, such as at Fig. 9C Element 912 is displayed at position 912c or 912d in the representation of the media item. For example, if the user's attention has corresponded to the second part of the representation of the media item for at least a threshold amount of time (e.g., 0.5 seconds, 1 second, 3 seconds, 7 seconds, 10 seconds, 20 seconds, or 30 seconds) and / or has moved less than a movement threshold (e.g., less than 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 100 cm, 300 cm, or 1000 cm of movement) after the user's attention has changed to point to the second part of the representation of the media item, the user interface element is displayed at the second position of the representation relative to the media item. In some embodiments, the computer system has detected that the user's gaze has changed from pointing to the lower right portion of the representation of the media item to pointing to the upper left portion of the representation of the media item, and / or has detected that the user's head has changed from being oriented toward the lower right portion of the representation of the media item to being oriented toward the upper left portion of the representation of the media item.

[0208] In some embodiments, in response to detecting that the attention of a user of the computer system has changed from being directed to a first portion of a representation of the media item to being directed to a second portion of the representation of the media item (1008a), based on determining that the user's attention does not satisfy the one or more criteria after changing to being directed to the second portion of the representation of the media item (for example, in some embodiments, if the attention (e.g., gaze) moves more than a threshold amount (e.g., moves more than 1 cm, 2 cm, 5 cm, 10 cm, 50 cm, 100 cm, 300 cm, or 1000 cm) after the user's attention changes to being directed to the second portion of the representation of the media item and / or the user's attention is not directed to the second portion of the representation of the media item for at least a threshold amount (e.g., 0.1 seconds, 1 second, 3 seconds, 7 seconds, 10 seconds, 20 seconds, or 30 seconds)), the computer system delays (1008c) displaying a user interface element at a second position relative to the representation of the media item corresponding to the second portion of the representation of the media item until the user's attention satisfies the one or more criteria, such as delaying until the one or more criteria are satisfied, and then displaying the user interface element at a second position relative to the representation of the media item corresponding to the second portion of the representation of the media item. Fig. 9C Element 912 is displayed at position 912c or 912d in the representation of the media item. For example, after the user's attention changes from the first part of the representation directed to the second part of the representation directed to the media item, the computer system delays displaying the user interface element at the second position relative to the representation of the media item until the user's attention points to the second part of the representation of the media item for at least the above-mentioned threshold amount of time (e.g., at a time after the computer system determines that the one or more criteria are met, the computer system determines that the one or more criteria have been met, and therefore displays the user interface element at the second position relative to the representation of the media item). In some embodiments, if the one or more criteria are not met, the user interface element continues to be displayed at the first position relative to the representation of the media item. In some embodiments, when the user's attention is removed from the first part of the representation of the media item, the user interface element fades out and does not fade in (e.g., at the second position relative to the representation of the media item) until the one or more criteria are met. Displaying or delaying displaying the user interface element after the user's attention changes enables the computer system to automatically display the user interface element when the user's attention has stabilized (e.g., stopped moving), and avoids the position of the user interface element from changing suddenly in response to the change in attention, thereby making it easier for the user to view the user interface element.

[0209] In some embodiments, when displaying the user interface element (1010a), before displaying the user interface element at the second position relative to the representation of the media item and while the user's attention is directed to a second portion of the representation of the media item, the computer system displays the user interface element at a third position relative to the representation of the media item that is different from the first position (such as at Fig. 9B The position of element 912 in Fig. 9C912 in the representation of the media item). For example, if the user's attention changes from the first part of the representation pointing to the media item at a first speed to the second part of the representation pointing to the media item, the computer system updates the position where the user interface element is displayed according to the change of the user's attention, but moves the user interface element at a second speed slower than the first speed. Therefore, in some embodiments, the computer system intentionally delays / reduces the movement of the user interface element so that the user interface element is not immediately displayed at the second position of the representation relative to the media item. For example, if the computer system is displaying the user interface element in the lower right portion of the representation of the media item before the user's attention changes to the second portion of the representation pointing to the media item, the computer system gradually moves the user interface element from the lower right portion of the representation of the media item to a position corresponding to the second portion of the representation of the media item (for example, the lower left portion of the representation of the media item) within a period of time (for example, 0.1 seconds, 0.3 seconds, 0.5 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, or 20 seconds). In some embodiments, the computer system displays the user interface element at the third position because the computer system is moving the user interface to a final position of the user interface element - the second position relative to the representation of the media item.

[0210] In some embodiments, when displaying a user interface element (1010a), after displaying the user interface element at a third position relative to the representation of the media item and while the user's attention is directed to a second portion of the representation of the media item, the computer system displays (1010c) the user interface element at a second position relative to the representation of the media item, including gradually moving the user interface element toward the second position after the user's attention begins to be directed to the second portion of the representation of the media item (e.g., within a corresponding time period greater than zero, such as 0.05 seconds, 0.1 seconds, 0.25 seconds, 0.5 seconds, 1 second, or 2 seconds), such as moving element 912 from Fig. 9B The position 912a or 912b gradually moves to Fig. 9C For example, after displaying the user interface at the third position relative to the representation of the media item, the computer system continues to move the user interface element toward the second portion of the representation of the media item at the second speed. Thus, when the user's attention stops changing / moving (e.g., such that one or more of the criteria described previously are met), the user interface element "catches up" with the user's attention.

[0211] Gradually moving the user interface element toward the second position relative to the representation of the media item avoids abrupt changes in the position of the user interface element in response to a change in attention, thereby making it easier for the user to view the user interface element.

[0212] In some embodiments, a first position relative to the media item corresponding to a first portion of the representation of the media item is offset from the first portion of the representation of the media item (1012a), such as position 912a and / or 912b from Fig. 9B For example, when the user's attention is directed to the first portion of the representation of the media item, the user interface element is displayed at an X position (e.g., a horizontal position relative to the user's viewpoint) that is offset from the X position of the first portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm), a Y position (e.g., a vertical position relative to the user's viewpoint) that is offset from the Y position of the first portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm), and / or a Z position (e.g., a distance relative to the user's viewpoint) that is offset from the Z position of the first portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm). In some embodiments, the Z position of the user interface element is based on the distance of the first portion of the corresponding media from the user's viewpoint, as described in method 800.

[0213] In some embodiments, a second position relative to the media item corresponding to a second portion of the representation of the media item is offset from the second portion of the representation of the media item (1012b), such as position 912c and / or 912d from Fig. 9C 908 in the representation of the media item. For example, when the user's attention is directed to the second portion of the representation of the media item, the user interface element is displayed at an X position (e.g., a horizontal position relative to the user's viewpoint) that is offset from the X position of the second portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm), a Y position (e.g., a vertical position relative to the user's viewpoint) that is offset from the Y position of the second portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm), and / or a Z position (e.g., a distance relative to the user's viewpoint) that is offset from the Z position of the second portion of the representation of the media item (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm). In some embodiments, the above-mentioned x and / or y offset distances correspond to a portion of the user's peripheral vision when the user's gaze is directed to the first portion / second portion.

[0214] Displaying the user interface element offset from the portion of the representation of the media item to which the user's attention is currently directed causes the computer system to automatically display the user interface element without obscuring the portion of the representation of the media item that the user is currently viewing.

[0215] In some embodiments, the user interface element includes subtitles (1014) for the media item being played by the computer system, such as 9A to 9E 912 in . In some embodiments, the corresponding user interface element includes a video caption that visually shows (e.g., in text form) the words spoken (e.g., by one or more characters and / or a narrator) at the current playback location in the representation of the media item. In some embodiments, the captions included in the user interface element are similar to the captions described in methods 800 and / or 1200.

[0216] Displaying subtitles at the user interface element causes the computer system to automatically present the media item in a manner that easily allows a deaf or hard of hearing user or a user who is watching on mute to view / interact with the media item.

[0217] In some embodiments, based on determining that the representation of the media item is at a first playback location, the subtitles for the media item include first content corresponding to the audio present at the first playback location (1016a), such as Fig. 9C 912. For example, the subtitles displayed in the user interface element include one or more words spoken by an actor / character at the first playback position, one or more words spoken by a narrator at the first playback position, and / or a text description of a sound event at the first playback position.

[0218] In some embodiments, based on determining that the representation of the media item is at a second playback location different from the first playback location, the captions for the media item include second content different from the first content corresponding to the audio present at the second playback location, such as Fig.9D 912 in the content of element 912. For example, as the playback positioning in the representation of the media items changes, the computer system updates the subtitles included in the user interface element. For example, when the representation of the media items is in the second playback positioning, the computer system optionally no longer displays the subtitles based on the spoken and / or non-spoken audio at the first playback positioning. On the contrary, the computer system optionally displays in the user interface element subtitles corresponding to the text description of the one or more words spoken by the actor / role at the second playback positioning, the one or more words spoken by the narrator at the second playback positioning and / or the sound event at the second playback positioning. In some embodiments, as the computer system plays the representation of the media items, the text color (e.g., font) of the subtitles does not change.

[0219] Changing subtitles displayed in a user interface element as the playback position of a representation of a media item changes causes a computer system to automatically present relevant information about the media item.

[0220] In some embodiments, while displaying a user interface element that includes subtitles for a media item and while the computer system is playing a representation of the media item, the computer system receives (1018a) via one or more input devices a first input corresponding to a request to stop displaying the user interface element, such as if device 101 receives (1018b) a request to stop displaying the user interface element. Fig. 9C 800). In some embodiments, the first input is received while the computer system is presenting a playback control user interface (e.g., a playback control user interface similar to that described in method 800). In some embodiments, the first input is directed to a first selectable option in the playback control user interface that, when selected (e.g., using an air gesture), causes the computer system to toggle off (e.g., stop displaying) subtitles for a representation of the media item (or toggle on subtitles for a representation of the media item if they are not currently being displayed).

[0221] In some embodiments, in response to receiving the first input, the computer system continues (1018b) playback of the representation of the media item without displaying the user interface element, such as if the device 101 would not display the user interface element while continuing to play the content 905. Fig.9D 912 in the element. For example, after the computer system detects user input directed to toggle off display of subtitles for a representation of a media item, the computer system no longer displays the user interface element during playback of the representation of the media item until the computer system detects a second input corresponding to a request to start displaying subtitles for the representation of the media item.

[0222] Ceasing display of a user interface element for at least a portion of a remainder of playback of a representation of a media item in response to user input provides an efficient way to manage display of information associated with a representation of a media item.

[0223] In some embodiments, the representation of the media item corresponds to a sporting event (e.g., based on the determination), and the information associated with the media item includes a (e.g., current) score (1020) of the sporting event, such as if element 912 includes score information. For example, if the computer system is displaying a sporting game, the user interface element includes information about the sporting game being played by the computer system, such as a score of the sporting game, teams participating in the sporting game, and / or a time remaining in the sporting game.

[0224] Displaying information about the sporting event when the representation of the media item corresponds to the sporting event causes the computer system to automatically display relevant information to a user of the computer system.

[0225] In some embodiments, based on determining that the representation of the media item is at a first playback position, the user interface element includes first information corresponding to the first playback position (1022a), such as Fig. 9C For example, when the computer system is presenting a representation of a media item at a first playback location, the user interface element includes information (e.g., names, biographies, facts, etc.) about the cast / characters at the first playback location (e.g., being displayed at the first playback location), about the music playing at the first playback location, and / or additional content (e.g., comments from the director of the media item regarding the first playback location).

[0226] In some embodiments, based on determining that the representation of the media item is at a second playback position different from the first playback position, the user interface element includes second information (1022b) that is different from the first information and corresponds to the second playback position, such as Fig.9D 912 in the representation of the media item. For example, as the playback location within the representation of the media item changes, the computer system updates the information included in the user interface element. For example, when the representation of the media item is in the second playback location, the computer system optionally no longer displays information based on the first playback location. Instead, the computer system optionally displays in the user interface element information (e.g., names, biographies, facts, etc.) about the cast / roles that are in the second playback location (e.g., being displayed at the second playback location), about the music played at the second playback location, and / or additional content (e.g., comments from the director of the media item about the second playback location).

[0227] Displaying information associated with the current playback position in the representation of the media item causes the computer system to automatically display relevant information to a user of the computer system.

[0228] In some embodiments, the user interface element is a playback control user interface (e.g., similar to the playback control user interface described in method 800), and the information associated with the media item includes one or more selectable options (1024) for modifying playback of a representation of the media item, such as Figure 7C726 in. For example, the position of the computer system display playback control user interface changes as the user's attention changes in one or more ways as described above. In some embodiments, the one or more optional options are similar to the one or more optional options of the playback control user interface previously described in method 800. In some embodiments, one or more optional options for modifying the playback of the representation of the media item include the optional option of making the computer system display the media item in the picture-in-picture presentation mode when selected. In some embodiments, one or more optional options for modifying the playback of the representation of the media item include the optional option of making the computer system present the content playback queue when selected. In some embodiments, one or more optional options for modifying the playback of the representation of the media item include the optional option of making the computer system rewind the representation of the media item to a predetermined amount (e.g., 5 seconds, 10 seconds, 15 seconds, 30 seconds or 60 seconds) when selected. In some embodiments, one or more optional options for modifying the playback of the representation of the media item include the optional option of making the computer system pause / play the representation of the media item when selected. In some embodiments, one or more optional options for modifying the playback of a representation of a media item include an optional option that causes the computer system to fast forward the representation of the media item by a predetermined amount (e.g., 5 seconds, 10 seconds, 15 seconds, 30 seconds, or 60 seconds) when selected. In some embodiments, one or more optional options for modifying the playback of a representation of a media item include an optional option that causes the computer system to present a subtitle / subtitle option associated with the representation of the media item when selected. In some embodiments, when the computer system receives an input including a user's gaze pointing to an optional option when performing a corresponding gesture at a predefined portion of the user (e.g., hand), the above optional option is selected. In some embodiments, the corresponding gesture is a pinch gesture in which the user touches the thumb of the hand to another finger of the hand. In some embodiments, the corresponding gesture is a tap gesture in which the user touches the finger of the hand to a position corresponding to the corresponding user interface element (e.g., an optional option or an input user interface element other than the optional option). In some embodiments, the selection input is an air gesture input.

[0229] Changing the position of the playback control user interface as the user's attention changes enables user interface elements to remain in the user's field of view without requiring additional user input.

[0230] In some embodiments, the first portion of the representation of the media item has a first position (e.g., a first X value) along a first axis (e.g., the X axis) and a second position (e.g., a first Y value) along a second axis (e.g., the Y axis) relative to the representation of the media item, and the first position of the user interface element is a third position (e.g., a second X value) along the first axis (e.g., the X axis) and a fourth position (e.g., a second Y value) along the second axis (e.g., the Y axis), the third position corresponding to the first position (1026a), such as relative to Fig. 9B 912b of element 912 in (e.g., and the fourth positioning is not based on and / or does not correspond to the second positioning). In some embodiments, the user interface element is displayed at an X position (e.g., a horizontal position relative to the user's viewpoint) and a predetermined Y position (e.g., a vertical position relative to the user's viewpoint), the X position being based on the X position of the portion of the representation of the media item to which the user's attention is directed. Thus, in some embodiments, the X position of the user interface element is variable (based on the user's attention), while the Y position of the user interface element is preselected / predetermined by the computer system (e.g., not based on the user's attention). For example, if a first portion of a representation of a media item is displayed at an X position such that the first portion is located in the right portion of the representation of the media item and at a Y position such that the first portion is located in the center of the representation of the media item (e.g., the first portion is in the right center portion of the representation of the media item), the user interface element is optionally displayed at an X position in the right portion of the representation of the media item (e.g., because the first portion is located in the right portion of the representation of the media item) and at a Y position directly above or directly below the first portion of the representation of the media item based on the predetermined / preselected Y position. In some embodiments, the Y position of the user interface element is variable (based on the user's attention), while the X position of the user interface element is preselected / predetermined (e.g., not based on the user's attention). For example, if a first portion of a representation of a media item is displayed at an X position such that the first portion is located in a right portion of the representation of the media item and at a Y position such that the first portion is located in the center of the representation of the media item (e.g., the first portion is in a right center portion of the representation of the media item), then the user interface element is optionally displayed at a Y position in the center portion of the representation of the media item (e.g., because the first portion is located in the center portion of the representation of the media item) and at an X position directly to the right or to the left of the first portion of the representation of the media item based on the predetermined / preselected X position.

[0231] In some embodiments, the second portion of the representation of the media item has a fifth position along the first axis that is different from the first position and a sixth position along the second axis that is different from the second position, and the second position of the user interface element is a seventh position along the first axis and a fourth position along the second axis, the seventh position corresponding to the fifth position (1026b), such as relative to Fig.9D 912 d of element 912 in (e.g., and the fourth positioning is not based on and / or does not correspond to the sixth positioning). For example, when the user's attention changes to be directed to the second portion of the representation of the media item (which is at a different X and / or Y position than the first portion of the representation of the media item), the computer system displays the user interface element at an X position based on the X position of the second portion of the representation of the media item and at a predetermined Y position (e.g., the same Y position as described above). For example, if the second portion of the representation of the media item is at an X position such that the second portion of the representation of the media item is located in a center portion of the representation of the media item (e.g., to the left of the first portion of the representation of the media item) and at a Y position such that the second portion of the representation of the media item is located in an upper portion of the representation of the media item (e.g., the second portion of the representation of the media item is located in an upper center portion of the representation of the media item), the computer system displays the user interface element at the X position that is in the center portion of the representation of the media item (e.g., because the second portion is located in the center portion of the representation of the media item) and at a Y position that is above or below the second portion of the representation of the media item based on a previously predetermined / preselected Y position (e.g., at the same Y position as compared to when the user interface element was displayed at the first position relative to the representation of the media item).

[0232] Displaying the user interface element at a variable position on the first axis and a fixed position on the second axis based on the positioning of the representation of the media item to which the user's attention is directed causes the computer system to display the user interface element at a predictable position.

[0233] In some embodiments, the representation of the media item satisfies one or more criteria, including when the representation of the media item corresponds to an immersive presentation of the media item (such as FIG. 9B to FIG. 9EIn some embodiments, when the representation of the media item corresponds to the immersive presentation mode, the representation of the media item has a length, width, and / or depth within which the content of the media item is displayed. In some embodiments, during immersive presentation of the media item, portions of the representation of the media item are optionally displayed closer to or farther from the user's viewpoint than in a non-immersive presentation mode (described below), and when the representation of the media item corresponds to a non-immersive presentation mode (such as Fig.9A The presentation of content 905 in (e.g., in some embodiments, when the representation of the media item corresponds to a non-immersive presentation mode, the representation of the media item occupies less than a threshold amount of the user's field of view (e.g., less than 75%, 60%, 50%, 40%, 30%, 20%, 10%, or 5% of the user's field of view)) does not satisfy the criteria, and displaying the user interface element at a first position corresponding to a first portion of the representation of the media item relative to the representation of the media item is based on determining that the representation of the media item satisfies the one or more criteria (1028a), such as in Fig. 9B Element 912 is displayed at position 912a or 912b in the immersive presentation mode. For example, when a representation of a media item is being presented in an immersive presentation mode, the user interface element is displayed at a location based on the user's attention as previously described. In some embodiments, while the user's attention is directed to a first portion of the representation of the media item and before the user's attention changes to a second portion of the representation of the media item, the second portion of the representation of the media item is not in the field of view from the user's viewpoint (e.g., not presented via a display generation component).

[0234] In some embodiments, when displaying a second representation of a second media item (e.g., the same as the media item but in a non-immersive presentation mode, or different from the media item and in a non-immersive presentation mode) via a display generation component, where the second representation of the second media item does not satisfy the one or more criteria, the computer system displays (1028b) a second user interface element at a predefined position relative to the second representation of the second media item, where the predefined position is not based on the user's attention, such as Fig.9A. For example, when the second media item is being presented in a non-immersive presentation mode, the user interface element is displayed in a bottom center portion of the second representation of the second media item, regardless of which portion of the second representation the user's attention is directed to. In some embodiments, in response to detecting a selection of a selection option in the playback control user interface, the second representation of the second media item is transitioned from a non-immersive presentation corresponding to the second media item to an immersive presentation corresponding to the second media item (as previously described in method 800).

[0235] Displaying the user interface based on the user's attention during the immersive presentation mode and displaying the user interface at a predetermined location during the non-immersive presentation mode enables the computer system to automatically display the user interface elements at the appropriate location when viewing content in different presentation modes.

[0236] In some embodiments, the first position and the second position relative to the representation of the media item are within the boundaries of the representation of the media item (1030a), such as Fig.9D ) (e.g., the user interface element is not displayed outside the X, Y, and / or Z boundaries of the representation of the media item and / or the user interface element is overlaid on at least a portion of the representation of the media item).

[0237] In some embodiments, while displaying the user interface element at the second position relative to the representation of the media item, the computer system detects (1030b) via one or more input devices that the user's attention changes from being directed toward the second portion of the representation of the media item to being directed outside the boundaries of the representation of the media item, such as Fig.9E 926 in . For example, the computer system has detected that the user's attention (e.g., gaze) is no longer directed to any part of the representation of the media item, but is directed to an area of ​​the displayed content outside the representation of the media item.

[0238] In some embodiments, after (e.g., in response to) detecting that the user's attention has changed from being directed at the second portion of the representation of the media item to being directed outside the boundaries of the representation of the media item, the computer system displays (1030c) a user interface element at a third position relative to the representation of the media item within the boundaries of the representation of the media item, such as Fig.9EElement 912 in is maintained in the border of content 905 as shown.In some embodiments, the third position relative to the representation of media items is the nearest part of the representation of media items outside the representation of media items where the user's attention currently points.For example, if the user's attention currently points to the upper right boundary of the representation of media items, the user interface element is optionally displayed at the position of the upper right boundary of the representation of media items near the representation of media items relative to the representation of media items.Therefore, in some embodiments, the user interface element can not be displayed at the position outside the X, Y and / or Z boundary defined by the representation of media items.In some embodiments, when the computer system receives the clear request (for example, the user selects the user interface element via pinching and dragging gestures and moves the user interface element to the boundary outside the representation of media items) outside the user's representation of media items, but not in response to the user's attention pointing to the representation of media items outside, the user interface element is optionally displayed at the position outside the boundary of the representation of media items.

[0239] Displaying user interface elements within the boundaries of a representation of a media item even when the user's attention is moved away from the representation of the media item provides an efficient way to relinquish display of user interface elements based on the user's attention when the user has stopped viewing the representation of the media item or has begun interacting with other content.

[0240] It should be understood that the specific order in which the operations in method 1000 are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0241] FIG. 11A to FIG. 11C Examples are shown of how electronic devices according to some embodiments of the present disclosure may display video subtitles with different visual appearances.

[0242] Fig.11A The three-dimensional environment 1130 visible via the display generation component 120 of the electronic device 101 and the top view 1114 of the three-dimensional environment 1130 are shown. Figures 1 to 6 As described above, the electronic device 101 optionally includes a display generation component 120 (eg, a touch screen) and a plurality of image sensors (eg, Figure 3The image sensor 314). The image sensor optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the electronic device 101 can use to capture one or more images of the user or a portion of the user when the user interacts with the electronic device 101. In some embodiments, the user interface shown and described below can also be implemented on a head-mounted display, which includes a display generation component that displays the user interface to the user, and sensors that detect the physical environment, movement of the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0243] In some embodiments, the three-dimensional environment 1130 includes one or more representations of physical objects in the physical environment 1132 of the electronic device 101 and / or one or more virtual objects (e.g., objects that are not located in the physical environment 1132 but are generated and displayed by the electronic device 101). Fig.11A In the example of FIG. 1 , the three-dimensional environment 1130 includes a representation 1110 of a table in the physical environment 1132 of the electronic device 101. The representation 1110 of the table is optionally a photo-realistic, simplified, or cartoon representation of the table in the physical environment of the electronic device 101, and is displayed in the three-dimensional environment 1130 in response to the electronic device 101 capturing one or more images of the table in the physical environment 1132. Alternatively, the representation 1110 of the table may be visible via a transparent or semi-transparent display of the electronic device 101 (e.g., the representation 1110 of the table is not displayed in the three-dimensional environment 1130).

[0244] In addition, if Fig.11A As shown, three-dimensional environment 1130 includes media user interface object 1104 (e.g., having one or more of the characteristics of media user interface objects 704 and / or 904) that is currently presenting a representation of media item 1105 at first playback location 1120. Fig.11A As shown in the overhead view 1114 in FIG, at the first playback position 1120, the representation of the media item 1105 includes actor A 1106 at a first position and a first depth in the representation of the media item 1105 and actor B 1108 at a second position (e.g., different from the first position) and a second depth (e.g., different from the first depth) in the representation of the media item 1105.

[0245] In some embodiments, electronic device 101 displays subtitles for a representation of media item 1105, such as Fig.11A The video caption user interface element 1112 shown in 7A to 7EIn some embodiments, the electronic device 101 begins displaying the video caption user interface element 712 described in the description above in response to the electronic device 101 detecting / determining that spoken and / or non-spoken audio is present at the first playback location 1120. Fig.11A In the video subtitles user interface element 1112. For example, in Fig.11A , the electronic device 101 begins displaying the video caption user interface element 1112 in response to the electronic device 101 determining that actor A 1106 is speaking at the first playback position 1120 (e.g., the speech audio of the representation of the media item 1105 corresponds to actor A 1106 at the first playback position 1120).

[0246] like Fig.11A As shown, the video caption user interface element 1112 includes text 1131 corresponding to spoken and / or non-spoken audio at the first playback position 1120 (e.g., corresponding to words spoken by actor A 1106 at the first playback position 1120) and a backplane element 1133 displayed behind the text 1131. In some embodiments, when audio (e.g., speech) is not presented at the first playback position 1120 in the representation of the media item 1105, the electronic device 101 does not display the video caption user interface element 1112.

[0247] Also like Fig.11A As shown, the electronic device 101 has detected that the attention of the user 1116 is currently directed toward the actor A 1106. In some embodiments, in response to detecting via the one or more image sensors 314 that the gaze 1134 of the user 1116 is directed toward the actor A 1106, and / or in response to detecting via the one or more image sensors 314 that the head of the user 1116 is oriented toward the actor A 1106, the electronic device 101 detects that the attention of the user 1116 is directed toward the actor A 1106.

[0248] In some embodiments, the electronic device 101 displays the video caption user interface element 1112 at an X position (e.g., horizontal position relative to the user's 1116 viewpoint), Y position (e.g., vertical position relative to the user's 1116 viewpoint), and / or Z position (e.g., distance relative to the user's 1116 viewpoint) corresponding to the X, Y, and / or Z position of the portion of the representation of the media item 1105 to which the user's 1116 attention is directed, such as described in more detail with reference to methods 800 and / or 1000. For example, in Fig.11A In the embodiment, the electronic device 101 detects that the user 1116's attention is directed to the actor A 1106 and Fig.11A The video caption user interface element 1112 is displayed at the X, Y, and Z positions indicated in Fig.11AThe X, Y, and Z positions of the video captions user interface element 1112 in the video captions user interface element 1112 correspond to actor A 1106 (e.g., the video captions user interface element 1112 has an X positioning that is offset from (to the right of) actor A 1106, a Y positioning that is offset from (offset downward from) actor A 1106, and a Z positioning that is 0.0 mm, 0.1 mm, 0.2 mm, 0.3 mm, 0.7 mm, 1 mm, or 2 mm, or 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, or 50 cm in front of actor A 1106).

[0249] It should be noted that although the above example shows a scenario in which the video caption user interface element 1112 is displayed below and to the right of (e.g., offset from) actor A 1106 in response to the electronic device 101 detecting that the user 1116's attention (e.g., gaze 1134) is directed toward actor A 1106, those skilled in the art should understand that the electronic device 101 may also display the video caption user interface element 1112 as being offset from or overlaid on actor A 1106 in any direction in response to the electronic device 101 detecting that the user 1116's attention is directed toward actor A 1106. 7A to 9E and / or one or more of the reasons described in methods 800 and / or 1000, the electronic device 101 may also Fig.11A It should be understood that, for similar reasons as described above (e.g., because the user's attention is directed to a portion of the content and / or because the portion of the content has a certain depth), the electronic device 101 optionally displays the video caption user interface element 1112 at the position indicated in FIG. Fig. 11B and Fig. 11C Video caption user interface element 1112 is displayed at the position shown in FIG.

[0250] In some embodiments, as described in detail in method 1200, the transparency of backplane element 1133 is determined based on the brightness of media item 1105 (and / or based on the portion of the representation of media item 1105 on which electronic device 101 is displaying video caption user interface element 1112). Fig.11A, the electronic device 101 is displaying a backplane element 1133 with no transparency (e.g., completely opaque) because the electronic device 101 has determined that the video subtitles user interface element 1112 is overlaying a very bright portion of the representation of the media item 1105 (e.g., a bottom portion of the representation of the media item 1105 corresponding to a maximum amount of brightness). In some embodiments, it is desirable to display a black backplane element 1133 with a small amount of transparency or no transparency when the video subtitles user interface element 1112 is overlaying a light-colored (e.g., bright) portion of the representation of the media item 1105 because this enables the electronic device 101 to display the video subtitles user interface element 1112 with sufficient contrast so that the user 1116 of the electronic device 101 can still read the text 1131 with a minimal or reduced amount of eye strain (e.g., Fig.11A The video caption user interface element 1112 and the bottom portion of the representation of the media item 1105 have a contrast ratio of at least 100:1, 200:1, 500:1, or 700:1).

[0251] In some embodiments, the electronic device 101 displays the text 1131 included in the video captions user interface element 1112 in a font color that reduces the visual contrast between the text 1131 and the portion of the representation of the media item 1105 that the video captions user interface element 1112 is overlaying (or more generally, the representation of the media item 1105). Fig.11A , the electronic device 101 is displaying text 1131 having a white font color because the electronic device 101 determines that the white text color complements the white hue of the bottom center portion of the representation of the media item 1105 (the portion of the representation of the media item 1105 over which the video subtitle user interface element 1112 is overlaid)—(e.g., the white font color causes the text 1131 and the bottom center portion of the representation of the media item 1105 to have a contrast ratio of less than 100:1, 200:1, 500:1, or 700:1) and / or because the white font color still allows a user 1116 of the electronic device 101 to read the text 1131 with a minimal or reduced amount of eyestrain (e.g., the white font color causes the backplane element 1133 and the text 1131 to have a contrast ratio of at least 100:1, 200:1, 500:1, or 700:1).

[0252] exist Fig. 11B , the electronic device 101 is no longer presenting a representation of the media item 1105 at the first playback position 1120, and is now presenting a representation of the media item 1105 at the second playback position 1124. Fig. 11B As shown, at the second playback position 1124, the electronic device 101 now plays the Fig.11AIn some embodiments, the video caption user interface element 1112 is displayed in a larger size and with more text. Fig.11A The electronic device 101 begins to display the video caption user interface element 1112 in a larger size and with more text than the video caption user interface element 1112 because the amount of spoken and / or non-spoken audio at the second playback position 1124 is higher than that at the first playback position 1120. For example, at the second playback position 1124, actor B 1108 optionally speaks more words than actor A 1106 at the first playback position 1120, and therefore requires more text than actor A 1106. Fig.11A In some embodiments, the electronic device 101 adjusts the size of the video caption user interface element 1112 to Fig. 11B Before the size indicated in , the electronic device 101 stops displaying the text 1131 being presented in the video caption user interface element 1112 at the first playback position 1120.

[0253] In addition, if Fig. 11B As shown, Fig.11A In comparison, the electronic device 101 is now displaying the backplane element 1133 with a higher transparency. In some embodiments, the electronic device 101 begins to display the backplane element 1133 with a higher transparency because the electronic device 101 determines that the brightness and / or hue (e.g., color) of the representation of the media item 1105 has dimmed by an amount between the first playback position 1120 and the second playback position 1124 such that the video subtitles user interface element 1112 can now obscure less of the bottom center portion of the representation of the media item 1105 (e.g., the portion of the representation of the media item 1105 that the video subtitles user interface element 1112 is covering), while also still displaying the video subtitles user interface element 1112 with sufficient visual contrast to allow the user 1116 of the electronic device 101 to read the text 1131 with minimal or reduced amounts of eye fatigue (e.g., even if the user 1116 of the electronic device 101 is viewing the text 1131). Fig. 11B The transparency of the backplate element 1133 in the image is increased. Fig. 11B The video caption user interface element 1112 and the bottom portion of the representation of the media item 1105 still have a contrast ratio of at least 100:1, 200:1, 500:1, or 700:1).

[0254] In addition, if Fig. 11B As shown, the electronic device 101 has updated the font color of the text 1131 in the video subtitle user interface element 1112. In some embodiments, the electronic device 101 changes the font color of the text 1131 from a white font color (such as Fig.11A ) is updated to a dark gray font color (as Fig. 11B As shown), the electronic device 101 determines that the gray font color is better than the white font color (as shown in FIG. Fig.11A 1) will have less visual contrast with the gray background of the representation of the media item 1105 at the second playback location 1124 and / or because the gray font color will still allow the user 1116 to read the text 1131 with a minimal or reduced amount of eyestrain (e.g., even with the gray font color, the backpanel element 1133 and the text 1131 have a contrast ratio of at least 100:1, 200:1, 500:1, or 700:1).

[0255] exist Fig. 11C , the electronic device 101 is now presenting a representation of the media item 1105 at the third playback position 1126, and the brightness and / or hue (e.g., color) of the bottom center portion of the representation of the media item 1105 has changed from Fig. 11B In some embodiments, when the electronic device 101 is displaying the video captions user interface element 1112 overlying a portion of the representation of the media item 1105 that has a brightness less than a threshold amount (e.g., a brightness level less than 0%, 1%, 2%, 3%, or 5%), the electronic device 101 does not present the backplane element 1133 ( Fig. 11B For example, Fig. 11C As shown, while electronic device 101 is displaying a representation of media item 1105 at third playback position 1126, electronic device 101 no longer displays backplane element 1133, even though electronic device 101 continues to display the text of user interface element 1112, because video caption user interface element 1112 is overlaid on a bottom center portion of the representation of media item 1105, which has a brightness amount that is less than the threshold brightness amount described above. In some embodiments, for similar reasons as previously described above, the font color of text 1131 is not displayed at the bottom center portion of the representation of media item 1105. Fig. 11B and Fig. 11C Change between.

[0256] FIG. 12A to FIG. 12F 1 is a flowchart illustrating a method 1200 for displaying video subtitles with different visual appearances according to some embodiments. In some embodiments, the method 1200 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, a smart phone, a wearable computer, or a head mounted device, the computer system including a display generating component (e.g., Figure 1 , Figure 3 and Figure 4In some embodiments, method 1200 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 1200 are optionally combined, and / or the order of some operations is optionally changed.

[0257] In some embodiments, method 1200 is performed at a computer system (e.g., electronic device 101) that communicates with a display generation component (e.g., 120) and one or more input devices (e.g., 314). In some embodiments, the computer system is the same or similar to the computer system described above with reference to method 800 and / or 1000. In some embodiments, the one or more input devices are the same or similar to the one or more input devices described above with reference to method 800 and / or 1000. In some embodiments, the display generation component is the same or similar to the display generation component described above with reference to method 800 and / or 1000.

[0258] In some embodiments, a representation of the media item at the first playback location is displayed via one or more display generation components, such as Fig.11A When the content 1105 in the corresponding media item is displayed (e.g., in some embodiments, the representation of the media item is the same as or similar to the representation of the media item described in methods 800 and / or 1000), the computer system displays (1202a) a user interface element (e.g., 1112) (e.g., such as the user interface element described in methods 800 and / or 1000) that includes information (such as text 1131) associated with the representation of the media item. In some embodiments, the user interface element includes video subtitles that visually show (e.g., in text form) words spoken (e.g., by one or more actors / roles) at a first playback position in the representation of the media item. In some embodiments, if there is no speech audio in the corresponding media item at the first playback position in the corresponding media item (e.g., no actor in the corresponding media item is speaking), the user interface is not displayed via the display generation component. In some embodiments, the corresponding user interface element includes two components: a backboard / motherboard and text (e.g., video subtitles) displayed on or in front of the backboard / motherboard.

[0259] In some embodiments, a user interface element is displayed over a first portion of a representation of a media item (1202b), such as relative to Fig.11A In some embodiments, the user interface element has a first visual appearance (e.g., as described in more detail later) (1202c) based on one or more visual characteristics (e.g., brightness, transparency, opacity, lightness, and / or average or dominant hue) of a first portion of a representation of the media item, such as relative to the first portion of the representation of the media item. Fig.11A 1131 and / or backplane element 1133 in the user interface element. In some embodiments, the first visual appearance corresponds to the visual appearance of the video subtitle text displayed in the user interface element, corresponds to the visual appearance of the backplane / mother of the user interface element, and / or corresponds to the opacity or transparency of the user interface element, as described in more detail below. In some embodiments, the user interface element is displayed with the first visual appearance because the first visual appearance reduces or minimizes the contrast between the first content and the backplane / mother of the user interface element. In some embodiments, the user interface element is displayed with the first visual appearance because the first visual appearance causes the user interface element to be displayed with sufficient contrast between the user interface element and the first portion of the representation of the media item (e.g., a contrast ratio of at least 100:1, 200:1, 500:1, or 700:1) and / or a color corresponding to the color of the content, so that the text included in the user interface element complements the content color of the first portion of the representation of the media item (as described in more detail below) while keeping the user readable with minimal or reduced eye fatigue.

[0260] In some embodiments, after displaying (1202d) via one or more display generation components a representation of the media item at a first playback position (wherein the user interface element is displayed over a first portion of the representation of the media item), the computer system displays (1202e) via the one or more display generation components a second playback position that is different from the first playback position (such as Fig. 11B In some embodiments, the playback position of the corresponding media is changed from the first playback position to the second playback position because the corresponding media item is currently being played at the computer system.

[0261] In some embodiments, when displaying a representation of the media item at the second playback position, the computer system displays (1202f) a user interface element over a second portion of the representation of the media item at the second playback position via one or more display generation components, wherein the user interface element is displayed with a second visual appearance that is different from the first visual appearance, wherein the second visual appearance is based on one or more visual characteristics (e.g., brightness, transparency, opacity, lightness, and / or average or dominant hue) of the second portion of the representation of the media item at the second playback position, such as Fig.11A Compared to Fig. 11BIn some embodiments, the visual appearance of the text 1131 and / or backboard element 1133 that are changed in the video playback. For example, as the computer system displays the representation of the media items at different playback locations, the visual appearance of the corresponding user interface element is based on the visual characteristics of the content currently being displayed at the media user interface object. In some embodiments, as described in more detail below, if the second content is darker than the first content, the backboard / motherboard of the corresponding user interface element is optionally more transparent than when the backboard / motherboard of the corresponding user interface element is displayed with the first visual appearance. In some embodiments, as described in more detail below, if the second content is darker than the first content, the color of the text in the corresponding user interface element is optionally darker than the color of the text when the corresponding user interface element is displayed with the first visual appearance. On the contrary, in some embodiments, if the second content is brighter than the first content, the color of the text in the corresponding user interface element is optionally brighter than the color of the text when the corresponding user interface element is displayed with the first visual appearance. In some embodiments, if the second content is brighter than the first content, then compared with when the backboard / motherboard of the corresponding user interface element is displayed with the first visual appearance, the backboard / motherboard of the corresponding user interface element is optionally more opaque. In some embodiments, as the media user interface object transitions from displaying the first content to displaying the second content, the corresponding user interface element increases or decreases in size, as will be described in more detail below. In some embodiments, the corresponding user interface element is displayed at a position closer to the user's viewpoint than the second content, so that the corresponding user interface element appears to be at least partially overlaid on the second content from the user's viewpoint. Modifying the visual appearance of the user interface element as the playback positioning within the representation of the media item changes provides an efficient way to reduce the visual contrast between the user interface element and the representation of the media item and to reduce the likelihood that the user interface element becomes visually distracting to the user of the computer system when viewing the representation of the media item, thereby enabling the user to use the computer system quickly and efficiently.

[0262] In some embodiments, displaying the user interface element with a first visual appearance includes displaying the user interface element with a first opacity amount, and displaying the user interface element with a second visual appearance includes displaying the user interface element with a second opacity amount different from the first opacity amount (1204), such as displaying the backplane element 1133 at a different opacity level. In some embodiments, the opacity of the user interface element is determined based on visual characteristics (e.g., brightness, transparency, opacity, lightness, and / or average or dominant hue) of the portion of the representation of the media item on which the user interface is overlaid. For example, when the computer system displays the user interface element overlaid on a second portion in the representation of the media item at a second playback position, the user interface element is optionally more opaque than when the computer system displays the user interface overlaid on the first portion in the representation of the media item at a first playback position because the second portion of the representation of the media item at the second playback position is brighter than the first portion of the representation of the media item at the first playback position. On the contrary, compared with when the computer system displays the user interface on the first part covered in the representation of the media item at the first playback positioning place, when the computer system displays the user interface element on the second part covered in the representation of the media item at the second ...

Claims

1. A method comprising: At a computer system in communication with one or more display generating components and one or more input devices: When displaying, via the one or more display generating components, a representation of a media item having content at different depths, displaying overlaid on a first portion of the representation of the media item a corresponding user interface element including subtitles for the media item being played by the computer system, wherein displaying overlaid on the first portion of the representation of the media item the corresponding user interface element comprises: Based on determining that the first portion of the representation of the media item is being presented at a first depth in the representation of the media item, displaying the corresponding user interface element at a first distance from a viewpoint of a user of the computer system; and Based on determining that the first portion of the representation of the media item is being presented at a second depth in the representation of the media item that is different from the first depth, displaying the corresponding user interface element at a second distance from the viewpoint of the user that is different from the first distance.

2. The method according to claim 1, further comprising: After presenting the first portion of the representation of the media item at the first depth in the representation of the media item, displaying the first portion of the representation of the media item at a third depth in the representation of the media item that is different from the first depth, and displaying the corresponding user interface element at a third distance from the user's viewpoint that is different from the first distance while displaying the first portion of the representation of the media item at the third depth in the representation of the media item.

3. The method of claim 1 , wherein when the corresponding user interface element is displayed as overlaying the first portion of the representation of the media item, the corresponding user interface element is displayed at the first distance from the viewpoint of the user, the method further comprising: After displaying the corresponding user interface element overlying the first portion of the representation of the media item at the first distance from the viewpoint of the user, displaying the corresponding user interface element overlying a second portion of the representation of the media item that is different from the first portion via the one or more display generation components, wherein displaying the corresponding user interface element overlying the second portion of the representation of the media item comprises: Based on determining that the second portion of the representation of the media item is being presented at a third depth in the representation of the media item that is different from the first depth, displaying the corresponding user interface element at a third distance from the viewpoint of the user that is different from the first distance.

4. A method according to claim 1, wherein when one or more criteria are met, the corresponding user interface element is overlaid on the first portion of the representation of the media item, and the one or more criteria include a criterion that is met when a target focal plane corresponds to the first portion of the representation of the media item, and when the one or more criteria are not met, the corresponding user interface element is not overlaid on the first portion of the representation of the media item.

5. The method of claim 1 , wherein displaying the representation of the media item comprises simultaneously displaying the first portion of the representation of the media item and a second portion of the representation of the media item that is different from the first portion, the method further comprising: displaying the corresponding user interface element overlying the first portion of the representation of the media item in response to determining that the first portion of the representation of the media item is closer to the viewpoint of the user than the second portion of the representation of the media item; as well as Based on determining that the second portion of the representation of the media item is closer to the viewpoint of the user than the first portion of the representation of the media item, the corresponding user interface element is displayed overlaid on the second portion of the representation of the media item.

6. The method according to claim 1, further comprising: receiving, via the one or more input devices, a first input directed to the representation of the media item while displaying the representation of the media item with the corresponding user interface element overlaying the first portion of the representation of the media item; as well as In response to receiving the first input: displaying, via the one or more display generating components, a playback control user interface comprising one or more selectable user interface elements for modifying playback of the representation of the media item; and Displaying the corresponding user interface element overlying the first portion of the representation of the media item is ceased.

7. The method of claim 6, wherein the playback control user interface is displayed when one or more criteria are met, and the playback control user interface is not displayed when the one or more criteria are not met, the method further comprising: after ceasing display of the corresponding user interface element and while displaying the playback control user interface, determining that the one or more criteria are not satisfied; as well as In response to determining that the one or more criteria are not satisfied: ceasing to display the playback control user interface via the one or more display generating components; as well as The respective user interface elements are displayed via the one or more display generating components.

8. The method according to claim 6, further comprising: In response to receiving the first input and while displaying the playback control user interface: The respective user interface element is displayed, via the display generation component, overlaid on a second portion of the representation of the media item that is different from the first portion.

9. The method according to claim 1, further comprising: detecting a change in a relative position of the representation of the media item with respect to the viewpoint of the user while displaying the corresponding user interface element overlying the first portion of the representation of the media item; as well as Upon detecting the change in the relative position of the representation of the media item with respect to the viewpoint of the user: Based on the change in the relative position of the representation of the media item with respect to the viewpoint of the user, the corresponding user interface element is moved from the first portion covering the representation of the media item to a second portion covering the representation of the media item that is different from the first portion.

10. A method according to claim 1, wherein the corresponding user interface element is overlaid on the first portion of the representation of the media item when one or more criteria are met, the one or more criteria including a criterion that is met when the audio of the media item corresponds to the first portion of the representation of the media item, and when the one or more criteria are not met, the corresponding user interface element is not overlaid on the first portion of the representation of the media item.

11. The method according to claim 1, wherein: Based on determining that the representation of the media item is at a first playback location, the subtitles for the media item include first content corresponding to audio present at the first playback location; and Based on determining that the representation of the media item is at a second playback location different from the first playback location, the subtitles for the media item include second content different from the first content corresponding to audio present at the second playback location.

12. The method according to claim 1, further comprising: receiving, via the one or more input devices, a first input corresponding to a request to stop displaying the user interface element while the corresponding user interface element is being displayed and while the computer system is playing the representation of the media item; and In response to receiving the first input, playback of the representation of the media item continues without displaying the corresponding user interface element.

13. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 12.

14. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 12.

Images