Techniques for presenting enlarged views in augmented reality environment

Abstract

The present disclosure relates to techniques for presenting enlarged views in an augmented reality environment. Embodiments disclosed herein relate to devices, systems, and methods for presenting enlarged views in an augmented reality environment. In particular, the magnified view includes a zoom reticle presented at a display location of the display. The zoom reticle includes magnified content including a magnified portion of a user's field of view. For example, the magnified content may be generated from image data selected from a corresponding portion of a field of view of the camera. The position of the zoom reticle on the display and the portion of the field of view that is magnified may vary in different situations, such as described herein.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 541,284, filed on September 28, 2023, under 35 U.S.C. §119(e), the contents of which are incorporated herein by reference as if fully disclosed herein. Technical Field

[0003] The described embodiments generally relate to presenting a magnified view in an extended reality environment. Background Art

[0004] Extended reality systems can be used to generate partially or completely simulated environments (e.g., virtual reality environments, mixed reality environments, etc.), where virtual content can replace or enhance the physical world. The simulated environment can provide an engaging experience for the user and is used in games, personal communications, virtual travel, health care, and many other scenarios. In some cases, the simulated environment may include information captured from the user's environment. This can provide users with additional ways to interact with their surroundings or learn more about their surroundings. Therefore, it may be desirable to extend the reality system to provide a user experience that facilitates such user interaction. Summary of the invention

[0005] Embodiments described herein relate to systems, devices, and methods for presenting a magnified view in an extended reality environment. Some embodiments relate to a method of operating an electronic device, the method comprising selecting a display position of a display of the electronic device. Selecting an area of ​​a field of view of a camera of the electronic device, and detecting text in the area. In response to detecting the text, selecting an image modification technique. Displaying a zoom graticule at the display position, wherein the zoom graticule includes an enlarged portion of the area modified according to the selected image modification technique.

[0006] Other embodiments relate to an electronic device including a camera having a field of view and a display. The electronic device includes a processor operably coupled to the camera and the display such that the processor is configured to select a display position for the display and to select an area of ​​the field of view of the camera of the electronic device. The processor may detect text in the area and, in response to detecting the text, select an image modification technique. The processor may display a zoom graticule at the display position, wherein the zoom graticule includes an enlarged portion of the area modified according to the selected image modification technique.

[0007] Additionally, some embodiments relate to a method of operating an electronic device, the method comprising receiving a request to activate a zoom reticle. In response to the request, selecting a display location of a display of the electronic device and an area of ​​a field of view of a camera of the electronic device. Determining a content type associated with the area, and selecting a set of display attributes based on the determined content type. Displaying a zoom reticle at the display location, wherein the zoom reticle includes an enlarged portion of the area displayed according to the selected set of display attributes.

[0008] Still other embodiments relate to an electronic device comprising a camera having a field of view and a display. The electronic device comprises a processor operably coupled to the camera and the display such that the processor is configured to select a display position for the display and to select an area of ​​the field of view of the camera of the electronic device. The processor may determine a type of content associated with the area and select a set of display attributes based on the determined type of content. The processor may display a zoom graticule at the display position, wherein the zoom graticule comprises an enlarged portion of the area displayed according to the selected set of display attributes.

[0009] Some embodiments relate to a method of operating an electronic device, the method comprising receiving a request to activate a zoom reticle. In response to the request, selecting a display position of a display of the electronic device. Detecting text in an area of ​​a field of view of a camera of the electronic device, and selecting a magnification level based on the detected text. Displaying a zoom reticle at the display position, wherein the zoom reticle includes a portion of the area magnified according to the magnification level.

[0010] Additional embodiments are directed to a method of operating an electronic device, the method comprising: collecting position information of a user's gaze within a first area of ​​a gaze field during a first time period. Detecting movement of the user's gaze to a second area of ​​the gaze field during a second time period after the first time period. In response to detecting movement of the user's gaze to the second area, using the position information collected during the first time period to select a display position of a display of the electronic device. Displaying a zoom graticule at the display position, wherein the zoom graticule includes an enlarged portion of the gaze field.

[0011] Some embodiments relate to an electronic device comprising a device configured to detect a gaze of a user within a gaze field and an eye tracker. The electronic device comprises a processor operably connected to a camera and a display such that the processor is configured to collect position information of a user's gaze within a first area of ​​the gaze field during a first time period. The processor may detect movement of the user's gaze to a second area of ​​the gaze field during a second time period after the first time period. In response to detecting movement of the user's gaze to the second area, the processor may select a display position of a display of the electronic device, wherein the display position is selected using the position information collected during the first time period. The processor may display a zoom graticule at the display position, wherein the zoom graticule includes an enlarged portion of the gaze field.

[0012] Still other embodiments relate to a method of operating an electronic device, the method comprising: collecting, during a first time period, information about a user's gaze position within a gaze field of view. Receiving a request to activate a zoom graticule, wherein the request has a request type. In response to receiving the request, selecting an analysis technique based on the request type and selecting a display position of a display of the electronic device. Selecting a display position based on an analysis of the collected information using the selected analysis technique. Displaying a zoom graticule at the selected display position, wherein the zoom graticule includes an enlarged portion of the gaze field of view.

[0013] Some embodiments are directed to a method of operating an electronic device, the method comprising receiving a request for a target input in a field of view of a camera of the electronic device, displaying a zoom reticle at a display of the electronic device, wherein the zoom reticle comprises an enlarged portion of the field of view of the camera of the electronic device, determining the target input using the enlarged portion of the field of view, and executing the request relative to the target input.

[0014] Other embodiments relate to an electronic device that includes a camera having a field of view and a display. The electronic device includes a processor that is operably coupled to the camera and the display, such that the processor is configured to receive a request for a target input in the field of view of the camera of the electronic device. The processor may display a zoom reticle at the display of the electronic device, wherein the zoom reticle includes an enlarged portion of the field of view of the camera of the electronic device. The processor determines the target input using the enlarged portion of the field of view, and may execute the request relative to the target input.

[0015] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the drawings and by study of the following descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure will be readily understood through the following detailed description in conjunction with the accompanying drawings, in which like reference numerals represent like structural elements and in which:

[0017] Figure 1 A block diagram of an example electronic device that may be used in the extended reality systems described herein is shown.

[0018] FIG. 2A to FIG. 2B Depicted are diagrams showing portions of a physical environment from the perspective of an electronic device such as described herein.

[0019] Figure 3A Depicted is a front view of a scene in which a magnified view may be displayed by an electronic device as described herein. Figure 3B Depicts a zoom reticle in which a zoom reticle is displayed Figure 3A Front view of the scene.

[0020] Figure 4 A method of selecting an initial display position of a zoom reticle based on the type of request for activating the zoom reticle is described.

[0021] Figure 5 Described is a method of selecting an initial display position of a zoom reticle when a user's gaze is used to request activation of the zoom reticle.

[0022] FIG. 6A to FIG. 6D Depicted Figure 3A A front view of a scene and illustrating how the zoom reticle can be positioned within the scene.

[0023] Figure 7 A method of displaying a zoom reticle including magnified content is described, wherein the magnified content is presented according to a selected set of display parameters.

[0024] Figure 8 A method of displaying a zoom reticle including magnified content including text modified according to an image modification technique is described.

[0025] Fig.9A and Fig. 9B Depicted Figure 3A A front view of a scene is shown, and illustrates how a zoom reticle can present magnified content including text.

[0026] Fig.10 A method of using a magnified view to select a target input associated with a request is depicted.

[0027] Fig.11 Depicted Figure 3A A front view of the scene is shown and illustrates how the zoom reticle can be used to select a target input.

[0028] It should be understood that the proportions and sizes (relative or absolute) of the various features and elements (and collections and groupings thereof) and the boundaries, spacings, and positional relationships presented therebetween are provided in the accompanying drawings solely to facilitate understanding of the various embodiments described herein and therefore may not necessarily be presented or illustrated to measure and are not intended to indicate any preference or requirement for the illustrated embodiments to exclude the embodiments described in conjunction therewith. DETAILED DESCRIPTION

[0029] Reference will now be made in detail to the representative embodiments illustrated in the accompanying drawings. It should be understood that the following description is not intended to limit the embodiments to a preferred embodiment. On the contrary, it is intended to cover substitutions, modifications and equivalents that may be included in the essence and scope of the described embodiments defined by the appended claims.

[0030] Embodiments disclosed herein relate to devices, systems, and methods for presenting a magnified view in an extended reality environment. Specifically, the magnified view includes a zoom graticule presented at a display position of a display. The zoom graticule includes magnified content, which includes an enlarged portion of a user's field of view. For example, the magnified content may be generated from image data that is selected from a corresponding portion of a camera's field of view. The position of the zoom graticule on the display and the portion of the field of view that is magnified may vary in different circumstances such as those described herein.

[0031] Reference below Figures 1 to 11 However, those skilled in the art will readily appreciate that the detailed description given herein with respect to these drawings is for illustrative purposes only and should not be construed as limiting.

[0032] The devices and methods described herein can be used as part of an extended reality system in which an extended reality environment is generated and displayed to a user. Various terms are used herein to describe the various extended reality systems and associated extended reality environments described herein. For example, as used herein, a "physical environment" is a portion of the physical / real world surrounding a user that the user can perceive and interact with without the assistance of an extended reality system described herein. For example, a physical environment can include a room or outdoor space in a building, as well as any person, animal, or object in that space (collectively referred to herein as a "real-world object"), such as plants, furniture, books, etc.

[0033] As used herein, an "extended reality environment" refers to a fully or partially simulated environment that a user can perceive and / or interact with using an extended reality system as described herein. In some cases, an extended reality environment may be a virtual reality environment, which refers to a fully simulated environment in which the user's physical environment is completely replaced by virtual content within the virtual reality environment. The virtual reality environment may not be dependent on the user's physical environment, and therefore may allow the user to perceive that they are in a different simulated location (e.g., standing at a beach when they are actually standing in a room of a building). The virtual reality environment may include virtual objects that the user can interact with (e.g., simulated objects that can be perceived by the user but do not actually exist in the physical environment).

[0034] In other cases, the extended reality environment can be a mixed reality environment, i.e., a fully or partially simulated environment in which virtual content can be presented together with a portion of the user's physical environment. Specifically, the mixed reality environment can include a reproduction (including direct reproduction and / or indirect reproduction) and / or a modified representation of one or more portions of the user's physical environment surrounding the extended reality system. In this way, the user can perceive a portion of his physical environment (directly or indirectly) through the mixed reality environment while still perceiving the virtual content.

[0035] As used herein, "reproduction" of a portion of a physical environment refers to a portion of an extended reality environment that recreates that portion of the physical environment within the extended reality environment. For example, an extended reality system may (such as in the case of an augmented reality system) have a transparent or translucent display, and may be configured to present virtual content on (one or more) transparent or translucent displays to form an extended reality environment. In these embodiments, a user may directly view portions of the physical environment that are not obscured by the presented virtual content through (one or more) transparent or translucent displays. Therefore, these portions of the physical environment that can be directly viewed by the user will be considered reproductions for the purposes of this application, and are referred to herein as "direct reproductions."

[0036] In other embodiments, the extended reality system includes (one or more) opaque displays so that the user cannot directly view the physical environment through the display. In some embodiments, the extended reality system may include one or more cameras capable of capturing images of the physical environment. The extended reality system can present a portion of these images to the user by displaying these images via an opaque display, so that the user indirectly views the physical environment via the displayed images. When presented to the user as part of an extended reality environment, the image (or its parts) is considered to be a reproduction for the purposes of the present application (also referred to herein as "indirect reproduction").

[0037] It should be understood that the captured images of the physical environment used to generate the indirect reproduction can undergo standard image processing operations, such as tone mapping, color balancing, and / or image sharpening, in an effort to match the indirect reproduction to the physical environment. Additionally, in some cases, an extended reality environment is displayed using foveated rendering, where different portions of the extended reality environment are rendered using different fidelity levels (e.g., image resolution) depending on the user's gaze direction. In these cases, for the purposes of this application, the portions of the reproduction rendered at a lower fidelity using these foveated rendering techniques are still considered to be a reproduction.

[0038] As used herein, a "modified representation" of a portion of a physical environment refers to a portion of an extended reality environment that is derived from the physical environment, but that intentionally obscures one or more aspects of the physical environment. While an indirect reproduction attempts to replicate a portion of a user's physical environment within the extended reality environment, a modified representation intentionally changes one or more visual aspects of a portion of the user's physical environment (e.g., using one or more visual effects, such as artificial blur). In this way, a modified representation of a portion of a user's physical environment may allow the user to perceive certain aspects of that portion of the physical environment while obscuring other aspects. In an example of artificial blur, the user may still be able to perceive the general shape and placement of real-world objects within the modified representation, but may not be able to perceive visual details of those objects that are visible in the physical environment. Where foveated rendering is used to display the extended reality environment, portions of the modified representation that are in peripheral areas of the extended reality environment (relative to the user's gaze) may be rendered at a lower fidelity using foveated rendering techniques.

[0039] The reproduction and / or modified representation of the user's physical environment can be used in various extended reality environments. For example, the extended reality system can be configured to operate in a "pass-through" mode, during which the extended reality system generates and presents an extended reality environment that includes a reproduction of a portion of the user's physical environment. This allows the user to indirectly view a portion of their physical environment, even if one or more components of the extended reality system (e.g., a display) may interfere with the user's ability to directly view the same portion of their physical environment. When the extended reality system operates in pass-through mode, some or all of the extended reality environment displayed to the user may include a reproduction of the user's physical environment. In some cases, in addition to the reproduction of the user's physical environment, one or more portions of the extended reality environment may include virtual content and / or a modified representation of the user's physical environment. Additionally or alternatively, virtual content (e.g., graphical elements of a graphical user interface, virtual objects) may be overlaid on the reproduced portion. In this way, the user may be able to perceive virtual content and its physical environment at the same time.

[0040] Generally, the extended reality systems described herein include an electronic device capable of capturing images and displaying the images to a user as part of an extended reality environment. Figure 1 A block diagram of an example electronic device 100 is depicted, which can be part of an extended reality system as described herein. The electronic device 100 can be configured to capture, process, and display images as part of an extended reality environment. In some implementations, the electronic device 100 is a handheld electronic device (e.g., a smart phone or tablet) configured to present an extended reality environment to a user. In some of these implementations, the handheld electronic device can be temporarily attached to an accessory that allows the user to wear the handheld electronic device. In other implementations, the electronic device 100 is a head mounted device (HMD) worn by the user.

[0041] In some embodiments, the electronic device 100 has a bus 102 capable of operatively connecting the I / O portion 104 to one or more computer processors 106 and memory 108, and includes circuits that interconnect components of the electronic device 100 and control communication between components. The I / O portion 104 includes various system components that can assist the operation of the electronic device 100. The electronic device 100 includes a set of displays 110 and a set of cameras 112. A set of cameras 112 can capture images of the user's physical environment (e.g., scene), and these images can be used when generating an extended reality environment that can include an enlarged view as described herein (e.g., as part of a pass-through mode). A set of displays 110 can display an extended reality environment so that the user can view the extended reality environment, thereby allowing the user to perceive its physical environment via the electronic device 100. A set of displays 110 may include a single display, or may include multiple displays (e.g., one display for each eye of the user). Each display in a set of displays can utilize any suitable display technology, and may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a light emitting diode (LED) display, a quantum dot light emitting diode (QLED) display, etc.

[0042] For the purpose of discussion, the techniques for presenting a magnified view are discussed herein with respect to a single display. However, it should be understood that the same techniques can be extended to additional displays of an extended reality system. For example, if the electronic device 100 includes multiple displays (e.g., a first display and a second display), a zoom graticule can be displayed on each display. In this way, a first zoom graticule can be displayed on a first display as part of an extended reality environment so that it can be viewed by a user's first eye (e.g., when the electronic device 100 is worn on the user's head or otherwise held near the user's head). Similarly, a second zoom graticule can be displayed on a second display as part of an extended reality environment so that it can be viewed by a user's second eye. The first zoom graticule and the second zoom graticule can be displayed at different corresponding positions on the first display and the second display, which can collectively allow the user to perceive a single zoom graticule as if it is positioned at a specific position within the extended reality environment. For example, these relative positions can be selected so that the zoom graticule appears as if it is positioned at a specific distance from the user.

[0043] Additionally, techniques for presenting a magnified view are discussed with respect to a single camera, however it should be understood that the same techniques may utilize multiple cameras. For example, where a magnified portion of a field of view (e.g., a camera's field of view) is presented in a zoom reticle, the magnified portion may be generated using image data from a single camera or multiple cameras (e.g., by fusing images captured from multiple cameras). Similarly, where zoom reticles are displayed on multiple displays, each zoom reticle may include magnified image data captured from a different camera (or camera group). For example, a first zoom reticle displayed on a first display may include a corresponding magnified portion generated from a first camera (or camera group), while a second zoom reticle displayed on a second display may include a magnified portion generated from a different camera (e.g., a second camera or camera group).

[0044] The memory 108 of the electronic device 100 may include one or more non-transitory computer-readable storage devices. These non-transitory computer-readable storage devices may be used to store computer-executable instructions that, when executed by one or more computer processors 106, may cause the computer processors to perform the processes described herein (such as the various techniques for presenting a magnified view described herein). In addition, the non-transitory computer-readable storage devices may be used to store images captured as part of a media capture event as described herein.

[0045] A computer-readable storage device may be any medium that may tangibly contain or store computer-executable instructions for use by or in conjunction with an instruction execution system, apparatus, and device (e.g., one or more processors 106). In some examples, the storage device is a transient computer-readable storage medium. In some examples, the storage device is a non-transient computer-readable storage medium. Non-transient computer-readable storage devices may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memory such as flash memory, solid-state drives, and the like.

[0046] The one or more processors 106 may include, for example, a processor, a microprocessor, a programmable logic array (PLA), a programmable array logic (PAL), a general array logic (GAL), a complex programmable logic device (CPLD), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any other programmable logic device (PLD) that can be configured to execute an operating system and applications for the electronic device 100 and facilitate the presentation of a magnified view in an extended reality environment as described herein.

[0047] Thus, any of the processes described herein may be stored as instructions on a non-transitory computer-readable storage device so that a processor may utilize these instructions to perform the various blocks of the processes described herein. Similarly, the apparatus described herein includes a memory (e.g., memory 108) and one or more processors (e.g., processor 106) operably coupled to the memory. One or more processors may receive instructions from the memory and be configured to execute these instructions to perform the various blocks of the processes described herein. Any of the processes described herein may be performed using the apparatus described herein as a method of presenting an enlarged view in an extended reality environment.

[0048] In some implementations, the electronic device 100 includes a set of depth sensors 114 (which may include a single depth sensor or a plurality of depth sensors), each of which is configured to calculate depth information for a portion of the environment in front of the electronic device 100. Specifically, each of the depth sensors in the set of depth sensors 114 may calculate depth information within a coverage field (e.g., the widest lateral range over which the depth sensor can provide depth information). In some cases, the coverage field of a depth sensor in the set of depth sensors 114 may at least partially overlap with the field of view of at least one camera in the set of cameras 112 (e.g., the spatial range of a scene that a camera can capture using the image sensor of the camera), thereby allowing the depth sensor to calculate depth information associated with the field of view of the set of cameras 112.

[0049] Information from the depth sensor can be used to calculate the distance between the depth sensor and various points in the environment surrounding the electronic device 100. In some cases, the depth information from the set of depth sensors 114 can be used to generate a depth map, as described herein. The depth information can be calculated in any suitable manner. In one non-limiting example, the depth sensor can utilize stereo imaging, in which two images are taken from different positions, and the distance (difference) between corresponding pixels in the two images can be used to calculate depth information. In another example, the depth sensor can utilize structured light imaging, whereby the depth sensor can image the scene while projecting a known illumination pattern (typically using infrared illumination) toward the scene, and then can see how the pattern is distorted by the scene to calculate depth information. In yet another example, the depth sensor can utilize time-of-flight sensing, which calculates depth based on the amount of time it takes for light (typically infrared) emitted from the depth sensor to return from the scene. A time-of-flight depth sensor may utilize direct time-of-flight or indirect time-of-flight, and may illuminate the entire coverage field at once, or may illuminate only a subset of the coverage field at a given time (e.g., via one or more spots, strips, or other patterns that may be fixed or may scan across the coverage field). Where the depth sensor utilizes infrared illumination, the infrared illumination may be used for a range of environmental conditions without being perceived by the user.

[0050] In some implementations, the electronic device 100 includes an eye tracker 116. The eye tracker 116 may be configured to determine the position of the user's eyes relative to the electronic device 100 (or specific components thereof). The eye tracker 116 may include any suitable hardware for identifying and locating the user's eyes, such as one or more cameras, depth sensors, combinations thereof, etc. In some cases, the eye tracker 116 may also be configured to detect the position / direction of the user's gaze. It should be understood that the eye tracker 116 may include a single module configured to determine the position of the user's eyes, or may include multiple units, each of which is configured to determine the position of the user's corresponding eye.

[0051] Additionally or alternatively, the electronic device 100 may include a set of sensors 118 capable of determining the motion and / or orientation of the electronic device 100 (or certain components thereof). Figure 1The illustrated set of sensors 118 includes an accelerometer 120 and a gyroscope 122. Information from the set of sensors 118 may be used to compute pose information for the electronic device 100 (or specific components thereof). This may allow the electronic device 100 to track its position and orientation as the electronic device 100 moves. Additionally or alternatively, information captured by the set of cameras 112 and / or the set of depth sensors 114 may be used to compute pose information for the electronic device 100 (or specific components thereof), such as by using simultaneous localization and mapping (SLAM) techniques.

[0052] In addition, the electronic device 100 may include a set of input mechanisms 124 (e.g., a touch screen, soft keys, a keyboard, a virtual keyboard, buttons, knobs, joysticks, switches, dials, combinations thereof, etc.) that a user may manipulate to interact with and provide input to the augmented reality system. The electronic device 100 may include a communication unit 126 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication technologies. The electronic device 100 is not limited to Figure 1 Rather, other or additional components (e.g., GPS sensors, directional sensors such as a compass, physiological sensors, microphones, speakers, haptic engines, etc.) may be included in a variety of configurations as needed.

[0053] When a user uses an electronic device 100 having a display, an eye tracker, and a camera as part of an extended reality system, these components can work together to monitor relevant portions of the user's physical environment. For example, Figure 2A It shows that the above Figure 1 A diagram of an example relationship between a camera field of view 200, a gaze field of view 202, and a display boundary 204 of the electronic device 100. The camera field of view 200 defines an area over which an image may be captured by at least one camera (e.g., camera 112) of the electronic device 100. Image data from the camera field of view 200 may be used to generate magnified content, as described in more detail herein.

[0054] The gaze field of view 202 may correspond to an area on which the user's gaze can be tracked. In some embodiments, this may correspond to an area on which the location of the user's gaze can be determined with desired accuracy. The gaze field of view 202 may represent all or a subset of the user's full gaze range for a given head position. As shown, the gaze field of view 202 may be a subset of the camera field of view 200, however, this is not required. In some cases, the gaze field of view 202 may be the same size as or larger than the camera field of view 200.

[0055] The display boundaries 204 may correspond to an area in which an enlarged view, such as those described herein, may be viewed by a user of the electronic device 100 (e.g., when the electronic device 100 is worn as a head mounted device). In some cases, the display boundaries 204 may correspond to the physical boundaries of the display 110 of the electronic device 100. In other cases, the display boundaries 204 may correspond to a subset of the display area of ​​the display 110. In these cases, the display 110 may be able to display a portion of the extended reality environment outside the display boundaries 204 (e.g., virtual content, an indirect reproduction of the user's physical environment, and / or a modified representation of the user's physical environment), but the electronic device 100 may only present the enlarged view within the display area. This may allow the extended reality system to limit the enlarged view to a specific area, which may provide an improved user experience.

[0056] In some cases, the display boundary 204 at least partially overlaps the camera field of view 200 and the gaze field of view 202, and as shown, the display boundary 204 can be smaller than both the camera field of view 200 and the gaze field of view 202. However, this is not required, and the display boundary 204 can be the same size as or larger than the camera field of view 200 and / or the gaze field of view 202. The display 110 can be at least partially transparent so that the physical environment can be viewed through the display boundary 204, and the physical environment can be viewed through any surrounding portion of the display if the display boundary 204 is smaller than the physical boundary of the display 110. In this way, graphical elements (such as the magnified view described herein) can be superimposed on the physical environment within the display boundary 204.

[0057] In some cases, the electronic device 100 may also include an object detection field of view (not shown). The object detection field of view may correspond to the boundaries of an environment in which the electronic device 100 can perceive and / or recognize objects. For example, the object detection field of view may correspond to the camera field of view 200, or a combination of the camera field of view and the corresponding field of view of one or more other sensors (e.g., one or more depth sensors 114). For example, the depth sensor 114 may have a wider field of view than the camera field of view 200, and the depth sensor 114 may be able to track objects outside the camera field of view 200. In these cases, the object detection field of view may be larger than the camera field of view 200.

[0058] Figure 2BThe camera field of view 200, the gaze field of view 202, and the display boundary 204 are shown as they relate to a user 208 of the electronic device 100. For context, the eye tracker 116, the display 110, and the camera 112 are also shown. As shown, the camera field of view 200 may correspond to the field of view of the camera 112. The gaze field of view 202 may correspond to the area in which the user's gaze can be detected by the eye tracker 116. This may be limited based on the constraints of the eye tracker 116 (such as its location, its capabilities, etc.). Figure 2B In the example shown, display boundary 204 corresponds to the position of display 110 relative to the user. Although not shown, display 110 may be supported by a frame or other support structure that positions display 110 at a particular distance from user 208 and at a particular position relative to user 208. This may determine display boundary 204 as viewed by user 208.

[0059] Figure 3A and Figure 3B A scene 300 is depicted as part of an extended reality environment that can be viewed by a user (e.g., using Figure 1 Scene 300 may represent a portion of the user's physical environment that is visible to the user at a given moment. Depending on the configuration of electronic device 100, the user may be able to directly perceive a portion of their physical environment (e.g., outside the boundaries of display 110 and any frame or other supporting structure holding display 110). In addition, the user may perceive a portion of their physical environment through display 110, either as a direct reproduction (in the case of a transparent or semi-transparent display 110) or an indirect reproduction (e.g., as part of a pass-through mode).

[0060] In some cases, it may be desirable for a user to be able to see a magnified portion of scene 300. This may allow a user to better visualize an area of ​​scene 300 without requiring the user to physically move (e.g., walk closer to an object of interest). For purposes of discussion, scene 300 is shown in FIG. Figure 3A302a-302c. Specifically, the first object 302a is depicted as a flower pot with flowers, the second object 302b is depicted as a clock, and the third object 302c is depicted as a frog. Each of the multiple objects 302a-302c can be a real-world object or a virtual object. For example, the first object 302a can be a real-world flower pot with flowers. In these cases, depending on where the first object 302a is positioned within the gaze field of view 202 of the electronic device, the user may be able to directly view the first object 302a (e.g., through a transparent or translucent display or outside the boundaries of the display 110 of the electronic device 100) or indirectly view the first object 302a (e.g., as an indirect representation of the object). In other variations, the first object 302a can be a virtual object (e.g., a simulated flower pot with flowers that does not actually exist in the physical environment). Therefore, in some cases it is possible that the zoom reticle can present enlarged content including enlarged virtual content.

[0061] In order to present the magnified view, the electronic device 100 may receive a request to activate the zoom reticle. Depending on the configuration of the electronic device 100, the request may be received under some predetermined conditions (e.g., a software application running on the electronic device 100 may automatically request the electronic device 100 to activate the zoom reticle when certain criteria are met, with appropriate user permission) or when a user gives a command to activate the zoom reticle by interacting with the electronic device 100 (or another part of the extended reality system incorporated into the electronic device 100). For example, the user may give the command by performing a predetermined action. The predetermined action may include performing a gesture, making a specific facial expression or eye movement, providing manual input (e.g., pressing a designated button, interacting with a touch screen) to the electronic device 100 or an accessory device associated with the electronic device 100 (e.g., a joystick, a mouse, etc.), giving a voice command, viewing a specific area of ​​the gaze field 202, etc.

[0062] After receiving a request to activate the zoom reticle, the electronic device 100 may select a target area 304 of the scene 300 that will be magnified as part of displaying the zoom reticle. In some cases, the target area 304 is selected from a portion of the gaze field of view 202. Because the magnified content can be derived from the image data from the camera, the target area 304 can also define a corresponding area of ​​the camera field of view 200. By selecting an area of ​​the field of view 200 of the camera 112, the electronic device 100 will be able to capture one or more images (e.g., image streams) of the physical environment corresponding to the target area 304. In some cases, the target area 304 can be a default area of ​​the gaze field of view 202. In these cases, the default area of ​​the gaze field of view 202 can be selected as the target area, regardless of the content of the scene 300 and regardless of how the user is interacting with the scene 300. In other variations, the target area 304 can be dynamically selected based on the content of the scene 300 and / or how the user is interacting with the scene 300.

[0063] The electronic device 100 may also select a display position that represents a portion of the display 110 of the electronic device 100 where the zoom graticule will be displayed. When the zoom graticule is first activated, the electronic device 100 selects an initial display position. The electronic device 100 may select the initial display position in any suitable manner as described herein. In some cases, the electronic device 100 may select a default initial display position (e.g., selected independently of the content of the scene 300 and / or how the user is interacting with the scene 300). In other cases, the electronic device 100 may select an initial display position based on the content of the scene 300 and / or how the user is interacting with the scene 300, such as described in more detail below.

[0064] In some variations, the selected target area 304 may be linked to the selected display position such that there is a predetermined relationship between the target area 304 and the display position. For example, in some variations, it may be desirable to position the zoom reticle at least partially over a portion of the scene 300 that is being magnified. This may allow a user to view the magnified portion of the scene 300 in the context of the surrounding area of ​​the scene 300. In these cases, the selection of the target area 304 may determine the selection of the display position (or vice versa), and the selected target area 304 and the display position will correspond to a common portion of the scene 300. Additionally, the relative position between the display position and the target area 304 may be adjusted to give a perspective view of the magnified portion of the scene presented at a particular depth relative to the zoom reticle.

[0065] In other variations, electronic device 100 may choose to zoom in on a portion of scene 300 that is within the field of view of camera 112 but outside display boundaries 204. In some of these cases, it may be desirable to position the zoom reticle as close as possible to target area 304 while still remaining within display boundaries 204. Thus, in these cases, the location of the selected target area 304 may be used to select a display position (e.g., to minimize the distance between the two).

[0066] After selecting the target area 304 and the initial display position, the electronic device 100 will display the zoom reticle 306 at the initial display position of the display 110 of the electronic device 100, such as Figure 3B . The display zoom graticule 306 includes magnified content 308, which represents an enlarged portion of the target area 304. Specifically, the magnified content 308 may be generated from image data, which is captured by the camera 112 of the electronic device 100. In some cases, the image data from the camera 112 may be digitally magnified (e.g., using a "digital zoom" operation). In these cases, image data from a portion of the camera's field of view (e.g., corresponding to the target area 304 of the scene) may be modified to generate an enlarged image. In some cases, as part of the digital operation, the electronic device 100 may scale up the image data to account for the loss of resolution that may occur when generating the enlarged image. Additionally or alternatively, the camera 112 may include an optical zoom capability that may change the field of view of the camera 112 to at least partially magnify the image captured by the camera 112.

[0067] In some variations, the magnified content 308 may be updated over time as the scene 300 changes (e.g., as objects move within the scene 300 and / or the user reorients the electronic device 100 relative to the scene 300 by moving or looking around). For example, as subsequent images are captured by the camera 112, the image data used to form the magnified content 308 may be updated. Specifically, the camera 112 may capture a stream of images, each of which is used to generate a different frame of the magnified content 308. In other words, the zoom graticule 306 may present a stream of images representing a magnified portion of the scene 300 that is updated as the scene 300 changes. Similarly, the display position of the zoom graticule 306 may change over time from an initial display position. Additionally or alternatively, the position of the target area 304 relative to the gaze field of view 202 and the camera field of view 200 may change over time to capture different portions of the gaze field of view. In other variations, while zoom reticle 306 is displayed, the display position of zoom reticle 306 and / or target area 304 may remain fixed over time.

[0068] As previously described, in some variations, it may be desirable to select an initial display position of zoom reticle 306 based on the content of scene 300 and / or the user's interaction with the scene. For example, in some variations of the extended reality systems described herein, position information about the user's gaze may be used to select an initial display position of zoom reticle 306. Specifically, the location of the user's gaze within gaze field of view 202 may be detected by eye tracker 116. This information about the user's gaze location (also referred to herein as "position information") may be collected and temporarily stored, and may be analyzed to select an initial display position of zoom reticle 306. For example, the position information may be temporarily stored in a data buffer that stores the position information for a predetermined amount of time (e.g., replacing the oldest data in the buffer with new data). In this way, at any given point in time, a certain amount of the user's gaze history is available for use.

[0069] The electronic device 100 may be configured to analyze a portion of the user's gaze history in order to select an initial display position for the zoom reticle 306. The user's gaze history may reflect the area of ​​the gaze field 202 that the user wants to zoom in on using the zoom reticle 306. For example, if the user happens to be looking at a particular object or area in the scene (e.g., Figure 3A and Figure 3B 300), the electronic device 100 may identify the object or area as an area of ​​interest. The images captured by the camera 112 may also be temporarily stored (e.g., in an image buffer) so that the electronic device 100 may associate the user's gaze history with the images captured by the camera 112, which may facilitate the identification of the area of ​​interest. The electronic device 100 selects an initial display position and / or a target area to be magnified based on the identified area of ​​interest. For example, the target area 304 may be selected so that the zoom graticule 306 includes an enlarged portion of the area of ​​interest. Similarly, the initial display position may be selected so that the zoom graticule 306 is positioned near the area of ​​interest or partially overlaps the area of ​​interest.

[0070] Thus, the user's gaze history can be analyzed to select an area of ​​interest in a scene. In some variations, the electronic device 100 can use different techniques to analyze the user's gaze history depending on what type of request is used to activate the zoom reticle. For example, Figure 4 A method 400 is shown for selecting an initial display position of a zoom reticle based on the type of request for activating the zoom reticle. These operations may be performed, for example, as described above with respect to Figures 1 to 3B The electronic device 100 in question is used for execution.

[0071] At box 402, position information of a user's gaze may be collected during a first time period. The position information may be detected, for example, by an eye tracker as discussed herein. Thus, the collected position information may represent a gaze history of the user during the first time period. At box 404, a request to activate a zoom graticule is received. The request to activate the zoom graticule may be received in any manner as described herein (e.g., automatically or by a user command). In addition, the request may be received at any time with respect to the first time period. For example, the request may be received at or after the end of the first time period, such that the collected position information includes only position information collected before the request is identified. For example, the first time period may represent a time window immediately before the request to activate the zoom graticule is received. In other cases, the request may be received during the first time period, such that the collected position information includes information collected before and after the request is identified.

[0072] Requests to activate the zoom reticle may have request types. Specifically, each request may be associated with a request type selected from a list of candidate request types. For example, when voice commands are used to request activation of the zoom reticle, these voice commands may be associated with one or more voice command types. Similarly, a user may gaze at a particular area of ​​the field of view to request activation of the zoom reticle, and these requests may be associated with one or more gaze-based request types. There may be request types associated with manual input, gestures, facial expressions, etc.

[0073] Each request type can be associated with a corresponding analysis technique. Each analysis technique involves a set of operations that can be applied to the collected position information to help select an initial display position for the zoom reticle. For example, depending on the analysis technique, the electronic device may select a specific subset of the position information, apply weights to specific portions of the position information, etc., in an attempt to determine where the user will want to position the zoom reticle. In other words, each analysis technique can be applied to the collected position information to identify an area of ​​interest. The identified area of ​​interest can be used to select an initial display position. In some variants, if the selected analysis technique cannot identify an area of ​​interest based on the collected position information, the electronic device may select a default initial display position.

[0074] For example, whenever a voice command associated with a voice command type is used to request activation of a zoom reticle, the electronic device may use a first analysis technique to select an initial display position. In these cases, the user may be most likely viewing the region of interest while giving the voice command. Thus, the first analysis technique may prioritize the location information collected while the user is speaking. The first analysis technique may apply a greater weight to the location information, or may select a specific subset of data that includes the location information.

[0075] Conversely, when a user activates the zoom reticle using a gaze-based request type, the user may need to look away from the area of ​​interest in order to initiate the request. Therefore, the gaze-based request type may be associated with a second technique for selecting an initial display position. In these cases, the second technique may reduce the importance of the position information collected while the user is initiating the request. Figure 5 , Fig. 6A and Figure 6B , this article describes examples of such analysis techniques.

[0076] At box 406, an analysis technique is selected based on the request type. At box 408, an initial display position of the display of the electronic device is selected based on the analysis of the collected position information using the selected analysis technique. Specifically, some or all of the position information collected during the first time period may be analyzed according to the selected analysis technique. The electronic device may use the analysis as an input for selecting the initial display position. For example, if the analysis can identify that the user is already focusing on the area of ​​interest, the electronic device may determine whether the area of ​​interest is still present in the scene. Therefore, the initial display position may be selected based on the current position of the area of ​​interest within the gaze field of view. Therefore, while the user is initially already viewing the first portion of the gaze field of view, a change in the scene may cause the zoom graticule to be displayed at a different position in the gaze field of view. At box 410, the zoom graticule is displayed at the selected display position, and the zoom graticule includes an enlarged portion of the gaze field of view.

[0077] In some cases, the region of interest is associated with an object present in the gaze field. Specifically, the selected analysis technique can be used to identify a target object in the gaze field. In these cases, the position of the target object (e.g., the current position of the target object) can be used to determine the initial display position. It should be understood that the selected analysis technique can also be used to select a target region of the gaze field. In the case where the selected analysis technique identifies a region of interest, the region of interest can be used to select a target region. For example, in some cases, the target region can be selected so that at least a portion of the target object is magnified in the zoom reticle.

[0078] Figure 5 An example of a method 500 for selecting an initial display position of a zoom reticle when using a user's gaze to request activation of the zoom reticle is shown. Figures 1 to 3BThe electronic device 100 discussed is performed. At box 502, during a first time period, position information of the user's gaze in a first area of ​​the gaze field of view is collected. The position information can be collected in any suitable manner as discussed herein (e.g., using an eye tracker). At box 504, during a second time period after the first time period, the user's gaze is detected to move to a second area of ​​the gaze field of view. The second area may be different from the first area and may represent a portion of the gaze field of view that the user can view to initiate a request. In some variations, once the user's gaze enters the second area, it is considered that a request to activate the zoom graticule is received. In other variations, the request to activate the zoom graticule is not considered to be received until one or more additional requirements are met (e.g., the user's gaze remains in the second area for a predetermined dwell time). It should be understood that the first time period and the second time period of method 500 may represent different parts of the first time period described relative to method 400.

[0079] In response to determining that the user's gaze has moved to the second area (and any additional requirements are met), an initial display position of a display of the electronic device is selected using the position information collected during the first time period at block 506. At block 508, a zoom reticle is displayed at the selected display position and includes an enlarged portion of the gaze field of view.

[0080] Fig. 6A and Figure 6B Shown in Figure 3A and Figure 3B In particular, in the context of the scenario 300 of FIG. Fig. 6A , the depicted portion of scene 300 represents the boundaries of gaze field 202 of electronic device 100. Gaze field 202 is divided into first region 600 and second region 602. Second region 602 represents an activation region of gaze field 202 associated with a request to activate the zoom reticle. When the user's gaze moves into second region 602 (and any other additional requirements that may be required are met), electronic device 100 may determine that the user is requesting activation of the zoom reticle. Although in Fig. 6A Only two areas are shown in FIG. 2 , but in some cases, if there are other activation areas in the gaze field 202 that can be associated with other functions of the electronic device 100, the first area 600 can be further subdivided. In addition, in some variations, the second area 602 can be positioned outside the display boundary 204, such as Fig. 6A In other variations, the second area 602 can be at least partially positioned within the display boundary 204 .

[0081] The electronic device 100 may track the user's gaze 610 within the gaze field 202, such as Fig. 6A612a-612d represent pauses during which the user's gaze 610 remains at a fixed position within the gaze field of view 202. As depicted, the user's gaze 610 begins at the first area 600 of the gaze field of view 202 (e.g., during a first time period as discussed with respect to block 502 of method 500). During this time, the user's gaze 610 is mostly focused on the third object 302c (e.g., the user's gaze 610 includes three pauses 612a-612c immediately adjacent to the third object 302c), and then moves to the second area 602 during a second time period.

[0082] Therefore, when the user's gaze is in the second area 602 (e.g., after the pause 612d), the electronic device 100 may receive a user's request to activate the zoom graticule. The electronic device 100 may use the position information of the user's gaze 610 collected during the first time period to select the initial display position of the zoom graticule. Specifically, the electronic device 100 may use the gaze information to identify the region of interest in the scene 300, and may select the initial display position so that it corresponds to the region of interest. Because the presence of the user's gaze 610 in the second area 602 is intended to activate the zoom graticule (as opposed to the user viewing the region of interest), the position information collected while the user's gaze 610 is in the second area may be ignored for the purpose of calculating the display position. Similarly, the movement in the first area 600 when the user's gaze 610 approaches the second area 602 may be similarly ignored or given a reduced weight because the movement may be less indicative of the region of interest.

[0083] In some variations, selecting the initial display position includes detecting a target object in the gaze field. Specifically, the position information collected during the first time period can be used to detect an area of ​​interest that includes the target object. For example, the analysis of the user's gaze 610 as shown in FIG3 can select the third object 302c as the target object. The electronic device 100 can use the position of the target object 302c in the gaze field 202 to select the initial display position.

[0084] In some examples, the electronic device may determine the current position of the target object in the gaze field 202. It is possible that when the user's gaze 610 moves to the second area 602, the target object may move within the scene 302, and / or the electronic device 100 moves relative to the scene, such that the position of the target object in the gaze field 202 changes. Therefore, by determining the current position of the target object in the gaze field 202, the electronic device 100 can account for any such movement.

[0085] In some variations, the initial display position may be selected so that the zoom reticle at least partially overlaps the target object. Figure 6B As shown, zoom graticule 306 is positioned at the display position such that zoom graticule 306 is at least partially positioned above target object 302c. Additionally, a target area (not shown) may be selected to correspond to the target object such that zoom graticule 306 includes magnified content 608 depicting an enlarged portion of the target object. For example, Figure 6B As shown, the third object 302c is magnified within the zoom reticle 306.

[0086] If the target object is at least partially located within the display boundary (e.g. Figure 6B In some embodiments, the zoom reticle 306 may be positioned to at least partially overlap the target object if the target object is within the display boundaries 204 shown. However, if the current position of the target object is outside the display boundaries 204, it may not be possible to position the zoom reticle 306 to overlap the target object. Therefore, in some variations, when the current position of the target object is outside the display boundaries, the initial display position may be selected as a default display position (e.g., centered within the display boundaries). In other variations, when the current position of the target object is outside the display boundaries, the initial display boundaries may be selected to minimize the distance between the zoom reticle and the target object.

[0087] When the current position of the target object is outside the display boundaries, in some cases, a target area of ​​the gaze field of view (e.g., which is magnified within the zoom reticle 306) may be selected to correspond to the initial display position. In these cases, the zoom reticle 306 will overlap with a portion of the scene 300 being magnified, but may not magnify the target object. In other variations, the target area of ​​the gaze field of view may be selected to correspond to the current position of the target object. In these cases, the target area may correspond to a portion of the gaze field of view 202 that is outside the display boundaries 204. Thus, the zoom reticle 306 may include a magnified portion of the target object, but may not be positioned to overlap with the target object.

[0088] When zoom reticle 306 is positioned at an initial display position, zoom reticle 306 may remain fixed at the display position or may move to one or more updated positions within display boundaries 204. For example, when the initial display position is selected based on a target object (or another area of ​​interest), electronic device 100 may be configured to track the movement of the target object in the gaze field. The movement may be the result of movement of the target object within scene 300 (e.g., when a frog jumps on a table) and / or movement of electronic device 100 relative to scene 300 (e.g., so that the gaze field overlaps with different portions of scene 300). Electronic device 100 may use the detected motion to select an updated display position and may move the zoom reticle to the updated display position. For example, Figure 6Cshows the third object 306c after it has moved to a new position within the gaze field 202. Figure 6B As shown, the electronic device 100 can update the position of the zoom graticule 306 to account for the movement, so that the third object 306c is still shown in the magnified content 308. In this way, the position of the zoom graticule 306 within the display boundary 204 can track the movement of the target object within the gaze field of view 202.

[0089] In some cases, the position of the zoom reticle 306 may be updated to track the movement of the target object only when the target object remains within a particular portion of the gaze field 202. For example, the position of the zoom reticle 306 is updated only when the target object remains within the gaze field 202. In these cases, the movement of the target object outside the gaze field 202 may be tracked (e.g., if the object detection field of the electronic device 100 is larger than the gaze field 202), but the electronic device 100 may not track the movement for the purpose of updating the zoom reticle 306. In some variations, the electronic device 100 may also be configured to move the zoom reticle 306 to a default display position if the target object leaves the gaze field 202. In other variations, the electronic device 100 may be configured to deactivate the zoom reticle 306 when the target object leaves the gaze field 202.

[0090] In other examples, the position of zoom reticle 306 is updated only when the target object remains within display boundary 204. For example, Fig.6D An example of scene 300 is shown in which a third object 302c has moved outside of display border 204. In this example, zoom reticle 360 ​​may remain at a position such as Figure 6C The same display position is shown even though the third object 302 c (eg, serving as a target object) has moved outside the display boundary 204 .

[0091] Additionally or alternatively, the target area of ​​the gaze field of view may be updated as the target object is detected to move. For example, the electronic device may select the target area and the display position so that a fixed relationship exists between the two (e.g., the zoom graticule 306 is positioned above a portion of the gaze field of view 202 that is being magnified). In these cases, when the zoom graticule 306 is positioned above the target object, the magnified content 308 may also include the magnified portion of the target object. In some cases, the target area may be updated to continue to at least partially overlap with the target object even if the display boundary 204 prevents further movement of the zoom graticule 306. For example, in some variations, when the target object (e.g., the third object 302c) is positioned above the gaze field of view 202, the target area may be updated to continue to at least partially overlap with the target object. Fig.6DWhen the display boundary 204 is shown as being moved outside, the target area can be selected as a portion of the fixation field of view 202 that is positioned outside the display boundary 204 to at least partially overlap with the target object. In this way, although the zoom reticle 306 remains within the display boundary 204, the magnified content 308 can still show a magnified portion of the target object.

[0092] Additionally or alternatively, the magnification level associated with the zoom graticule 306 may change as the target object moves relative to the gaze field of view 202. For example, when the zoom graticule 306 is initially displayed, the zoom graticule 306 may include magnified content 308 that is magnified at the initial magnification level. For example, a 2x magnification level would represent a situation in which a magnified portion of a scene (e.g., a target area) appears to be twice as large as the same portion of the scene when the zoom graticule 306 is not present. The initial magnification level may be a default magnification level (which may be modified based on user preferences), or may be selected based on the size or other characteristics of the target object. In some cases, the user may provide input to the electronic device 100 to update the magnification level.

[0093] As the target object moves within the gaze field 202 (e.g., by movement of the target object relative to the scene 300 and / or movement of the electronic device 100 relative to the scene 300), the target object may be closer to or further away from the electronic device 100. This may change how large the target object appears to the user (e.g., changing the size of the target object within the gaze field 202), and similarly change how large the target object appears in the zoom reticle 306. Accordingly, the magnification level may be updated to account for these changes. For example, the magnification level may increase as the target object moves away from the electronic device 100, or may decrease as the target object moves closer to the electronic device 100. This may cause the target object to appear at the same size within the zoom reticle 306, even though the target object is moving relative to the gaze field 202.

[0094] As previously described, the magnified content 308 may be generated from image data captured by one or more cameras (e.g., camera 112) of the electronic device 100. The image data may be processed in a manner that allows the magnified content 308 to be displayed according to a set of display properties. For example, it may be desirable to display the magnified content 308 at a particular brightness level, contrast level, transparency level, magnification level, combinations thereof, and the like. Thus, the set of display settings may include corresponding settings for any or all of these properties, and the electronic device 100 may process the image data from the camera 112 to implement the set of display settings.

[0095] For example, the set of display settings may include contrast settings. In these cases, the contrast settings may define a desired contrast level for the magnified content 308 when it is displayed. The contrast of the image data may be adjusted as needed (e.g., using known image processing techniques that will be readily understood by those of ordinary skill in the art) to achieve the desired contrast level. Additionally or alternatively, the set of display properties may include a brightness setting (e.g., setting a desired brightness level for the magnified content 308), a transparency level (e.g., controlling how much of the underlying scene 300 is visible through the magnified content 308), a magnification level (e.g., setting a magnification level for the magnified content 308), a combination thereof, and the like. In general, the electronic device 100 may use these display settings to generate the magnified content 308 from a target area of ​​the gaze field 202.

[0096] In some cases, it may be desirable to set different display settings based on the content of the scene being magnified. Different display settings may be useful in different contexts to better allow the user to perceive the magnified content. For example, when using a zoom reticle to magnify text, it may be desirable to present the magnified content in a manner that improves the readability of the text. If the zoom reticle is used to magnify a particular type of object (e.g., a small animal), it may be desirable to present the magnified content in a manner that allows the user to see the fine details of the object and / or reduces the importance of any background content that is also visible in the zoom reticle. If the zoom reticle is used to magnify a work of art, it may be desirable to present the magnified content in a manner that makes the work of art appear more vivid (e.g., by adjusting the color saturation of the image data used to generate the magnified content).

[0097] Figure 7 An example of a method 700 for using an electronic device to display a zoom graticule and generate magnified content using content-specific display attributes is shown. These operations may be performed, for example, as described above with respect to Figures 1 to 3B 100. At block 702, the electronic device: i) selects a display position for a display of the electronic device, and also ii) selects a target area for a field of view of a camera of the electronic device. In some variants, block 702 may be performed in response to receiving a request to activate a zoom reticle, in which case the selected display position is an initial display position. The display position and target area may be selected in any manner as described herein.

[0098] At box 704, identify the content type associated with the target area. The content type is used as a classification of the subject present in the target area. In some variations, the content type is selected from a candidate content type list. As an example, an electronic device can distinguish between a target area containing text and those areas that do not contain text. In these cases, the candidate content type list may include one or more text content types (e.g., where text is identified in the target area) and one or more non-text content types (e.g., where text is not identified in the target area). In some of these cases, the candidate content type list may include multiple text content types (e.g., one or more text content types associated with labels (such as food labels or drug labels), one or more text content types associated with books or magazines, etc.). Additionally or alternatively, the candidate content type list may include multiple non-text content types (e.g., one or more non-text content types associated with artworks, such as paintings, sculptures, etc., one or more non-text content types associated with objects, such as animals, food, etc., etc.).

[0099] Any combination of the electronic device's camera and other sensors (e.g., the camera 112 and depth sensor 114 of the electronic device 100) may be used to determine the content type. For example, one or more image processing techniques (e.g., optical character recognition techniques, image classification techniques, combinations thereof, etc.) may be used to select the content type. These image processing techniques may be applied to images and sensor data associated with the target area, and in some cases may also be applied to images and sensor data associated with areas surrounding the target area. These surrounding areas may provide additional context to what content is present in the target area.

[0100] At block 706, a set of display attributes is selected based on the determined content type. For example, each content type in the list of candidate content types can be associated with a corresponding set of display attributes. The display attributes associated with a given content type can be fixed, or can be adjustable by a user. For example, a user can select specific display settings for a particular content type (e.g., a specific filter to be applied when a given content type (a non-text content type associated with a work of art) is identified).

[0101] At block 708, a zoom graticule is displayed at the selected display position, and the zoom graticule includes an enlarged portion of the target area displayed according to the selected set of display properties. In some cases, the enlarged portion of the target area fills the entire zoom graticule. In other cases, different portions of the zoom graticule may include magnified content displayed according to different sets of display properties. Specifically, the zoom graticule may be displayed such that an enlarged portion of the target area (e.g., a portion displayed according to the selected set of display properties) is displayed in a first area of ​​the zoom graticule. The second area of ​​the zoom graticule may also include an additional enlarged portion of the target area. The additional enlarged portion may be displayed according to a different set of display properties.

[0102] In some cases, the first area can be selected based on the determined content type. For example, if the content type is associated with a particular object (e.g., an animal), it may be desirable to display the object differently from any surrounding content. If the identified object does not completely fill the target area, the target area may be divided into a first portion corresponding to the object and a second portion that does not correspond to the object (e.g., instead corresponds to background content). The electronic device may select a first set of display attributes corresponding to the first area and a different second set of display attributes corresponding to the second area. Thus, the zoom graticule may: i) display a first magnified portion (e.g., corresponding to the object) in a first area of ​​the zoom graticule according to the first set of display attributes, and ii) display a second magnified portion (e.g., corresponding to background content) in a second area of ​​the zoom graticule according to the second set of display attributes.

[0103] When the zoom graticule is actively displayed, the set of display parameters may remain fixed or may be updated as the scene changes. For example, if the content of the scene changes (e.g., one or more objects move within the scene and / or the electronic device moves relative to the scene), the type of content associated with the target area may change. If the electronic device determines that the type of content associated with the target area has changed, the electronic device may select a new set of display attributes based on the current content type so that the zoom graticule includes an enlarged portion of the target area displayed according to the newly selected set of display attributes. In some cases, when the electronic device selects a new set of display attributes, the electronic device may be configured to gradually transition between these attributes so that the user does not perceive an abrupt change in the process used to generate the enlarged content.

[0104] In general, in these cases, for each frame to be displayed within the zoom reticle, the electronic device may: i) select a display location for the display and an area of ​​the camera's field of view, ii) determine a content type associated with the selected area, iii) select a set of display attributes based on the determined content type, and iv) display the zoom reticle at the display location, wherein the zoom reticle includes an enlarged portion of the target area displayed according to the selected set of display attributes. When the determined content type changes between frames, the selected set of display attributes also changes.

[0105] As previously mentioned, when enlarging text, it may be desirable to present the text in a manner that makes it easier for a user to read the text. Therefore, it may be desirable to apply one or more image modification techniques to image data that includes text. These image modification techniques may improve readability when the enlarged content is presented to the user. Figure 8 A method 800 of displaying a zoom reticle including magnified content is depicted, wherein the magnified content includes text modified according to an image modification technique. These operations may be performed, for example, as described above with respect to Figures 1 to 3B 802. The electronic device 100 discussed herein may be performed. At block 802, the electronic device: i) selects a display position for a display of the electronic device, and also ii) selects a target area for a field of view of a camera of the electronic device. In some variations, block 802 may be performed in response to receiving a request to activate a zoom reticle, in which case the selected display position is an initial display position. The display position and target area may be selected in any manner as described herein.

[0106] At box 804, text is detected in the target area. Text can be detected in any suitable manner, such as using optical character recognition technology. In response to detecting text, an image modification technique can be selected at box 806. Image modification techniques involve one or more image processing operations applied to the target area to modify its appearance when the target area is enlarged. Image modification techniques can be selected in any suitable manner. In some cases, image modification techniques can be selected based on the text itself and / or based on the type of content associated with the text, such as described above. In these cases, the image modification technique can be selected so that the enlarged content is displayed according to one or more display attributes as discussed above.

[0107] In some variations, the image modification technique includes contrast adjustment. Thus, contrast adjustment can be applied to image data from a target area to adjust the contrast of the text. Additionally or alternatively, the image modification technique may include brightness adjustment, which can be applied to image data from a target area to adjust the brightness of the text. In other variations, the image modification technique includes replacing image data from a camera with synthetic image content. For example, if the detected text includes specific words, the electronic device may generate virtual content that presents the same words. In these cases, the image data corresponding to the text can be replaced with virtual content. This may be beneficial in cases where the actual text in the scene may still be difficult to read by the user even after being enlarged and adjusted using other image modification operations. Presenting synthetic image content allows precise control over how the text is presented to the user.

[0108] At box 808, a zoom graticule is displayed at a display position, and the zoom graticule includes an enlarged portion of the target area modified according to the selected image modification technique. In some variations, the enlarged portion of the target area is presented at a magnification level selected based on the detected text. For example, it may be desirable to present the text at a target character size. The target character size may be a default size, or may be set for a specific user based on the preferences and / or optical prescription of a specific user. Therefore, when text is detected in the target area, the character size may be determined for the detected text. The character size may represent the size of the detected text, and may represent the size of the smallest detected carrier, the average size of all detected text, etc. The electronic device may select the magnification level based on the character size so that the text will have the target character size when enlarged in the zoom graticule.

[0109] In some variations, the image modification technique is applied only to a portion of the magnified content presented in the zoom graticule. Specifically, the zoom graticule may be displayed such that an enlarged portion of the text (e.g., a portion modified according to a selected image modification technique) is displayed in a first sub-region of the zoom graticule. A second sub-region of the zoom graticule may also include additional enlarged portions of the target area that are not modified according to the selected image modification technique. It should be understood that other image modification techniques may be applied to the second sub-region such that the second sub-region is presented according to a specific set of display attributes as discussed above. In this way, areas within the target region that include text may be processed to facilitate readability of the text, while other areas within the target region may be processed based on content present in those areas.

[0110] Fig.9A and Fig. 9B Shown in Figure 3A and Figure 3BIn particular, in the context of the scenario 300 of FIG. Fig.9A In FIG. 1 , a depicted portion of the scene 300 is contained within the camera field of view 200 of the electronic device 100. As shown, a target area 900 including text (in this example, including the time displayed on the second object 302b) is selected in the camera field of view 200. Fig. 9B In the embodiment of the present invention, a zoom graticule 906 is presented at the display position, and the zoom graticule may include magnified content 908 corresponding to the target area 900. At least a portion of the magnified content 908 (e.g., corresponding to the detected text) may be modified according to the selected image modification technique, such as relative to Figure 8 The method described in Fig. 9B In the example shown, the first sub-region 910 of the magnified content 908 is replaced with the synthetic image content. In these cases, the zoom graticule 906 may show the time currently displayed on the second object 302b, but the synthetic image content may be used to change the font, color, style, typographic emphasis and / or other attributes of the text. Although only the first sub-region 910 is modified according to the selected image modification technique, in other cases, the entire magnified content 908 may be modified according to the selected image modification technique.

[0111] In some cases, an enlarged view as described herein can be used to help facilitate the selection of a target input from an extended reality environment. As part of an extended reality environment, an electronic device can be configured to perform an action on behalf of a user that requires a target input (such as a target object) to be selected from the extended reality environment. Before the action can be performed, the target input must be identified. However, it may not be clear which part of the user's environment is intended to be the target input. Therefore, it may be useful to clarify the user's intent. Therefore, an enlarged view can be used to clarify the user's intent, and specifically to identify one or more target inputs.

[0112] For example, a user may initiate a request to an electronic device related to a target input in a physical environment. In particular, the request may be a voice command, such as a phrase "tell me about that," where "that" refers to an object in the physical environment that the user is looking at. In addition to a request for information, the request may be a command such as "turn this off," a reminder such as "next time I see this, remind me to put it away," or any other type of request. As another example, a user may point to an object in a physical environment. As yet another example, a user may look at and / or focus on an object in a physical environment. The request may be a one-time request such as those discussed above, or a standing request that is executed whenever a certain event occurs. For example, a user may request "whenever I see this, remind me to call Bill."

[0113] For some requests, the electronic device may be able to easily identify the target input based on the extended reality environment and / or the user's interaction with the extended reality environment. For example, in some cases, the electronic device may be configured to identify one or more candidate objects that are at least partially located in the gaze area at or about the time of the request. In the case where there is only one candidate object in the gaze area, or when the context or other information of the request provides a clear indication of the target input, the request can be performed relative to the target object. Following the above example, the electronic device can identify the target input as a plant that the user is watching and provide the user with information about the plant (e.g., the size of the plant, the watering status of the plant, care instructions for the plant, information about the species of the plant, etc.). However, in many cases, the gaze area may include multiple candidate objects, and it may not be clear which of the candidate objects is intended to be the target input.

[0114] Therefore, in order to facilitate the selection of a target object from the set of candidate objects, a zoom graticule may be displayed at the display of the electronic device, which may allow the user to select a target input. Fig.10 A method 1000 of using a magnified view to assist in selecting a target input associated with a request is depicted. At block 1002, an electronic device receives a request for a target input in the field of view of a camera of the electronic device. The request may be received in any suitable manner. Depending on the configuration of the electronic device 100, the request may be received under some predetermined conditions (e.g., a software application running on the electronic device 100 may automatically request the electronic device 100 to initiate an action requiring a target input with appropriate user permission) or when a user gives a command to initiate a request by interacting with the electronic device 100 (or another part of an extended reality system incorporated into the electronic device 100). For example, a user may give a command by performing a predetermined action. The predetermined action may include giving a voice command, performing a gesture, making a specific facial expression or eye movement, providing a manual input (e.g., pressing a specified button, interacting with a touch screen) to the electronic device 100 or an accessory device associated with the electronic device 100 (e.g., a joystick, a mouse, etc.), viewing a specific area of ​​the gaze field 202, etc.

[0115] The request may be associated with one or more actions that require the target input, and once the target input is recognized, the electronic device may use the target input to perform the actions. For example, the request may be a request to provide information related to the target input (e.g., a voice command such as "What's that?"), set a reminder related to the target input, control another device (such as a smart device) selected as the target input, etc.

[0116] At box 1004, a zoom graticule is displayed at a display of the electronic device, and the zoom graticule includes an enlarged portion of the field of view of the camera. In some cases, the zoom graticule can be activated first, and then a request is received at box 1002, so that the zoom graticule is already displayed when the request is received. In other cases, the zoom graticule is activated in response to receiving a request at box 1002. When the zoom graticule is activated in this manner, the zoom graticule can select an initial display position and an initial target area in any manner as described herein. For example, these parameters can be selected as default values, or can be based on the user's gaze history before receiving the request. In some cases, the selection of the initial display position and the initial target area can depend at least in part on the type of request received. For example, some requests requiring target input may cause the zoom graticule to be initially positioned at a default display position, while other requests requiring target input may be initially positioned at a display position selected based on the user's gaze history.

[0117] In some of these variations, the electronic device may attempt to identify the target input before activating the zoom reticle, and may activate the zoom reticle only if the electronic device is unable to identify the target input. If the electronic device is able to identify the target input before activating the zoom reticle, the electronic device may execute the request with respect to the target input without activating the zoom reticle.

[0118] At block 1006, a target input is determined using the magnified portion of the field of view presented in the zoom reticle. In some variations, this includes using electronics to detect user interaction with the zoom reticle. A target input may be selected based on the detected user interaction. This allows the user to actively control the selection of a target input by interacting with the zoom reticle.

[0119] In some variations, the location of a user's gaze within the zoom reticle may be used to select a target input. For example, an eye tracker (e.g., eye tracker 116 of electronic device 100) may determine the location of a user's gaze within a gaze field of view. While the zoom reticle is displayed, the eye tracker may determine the location of a user's gaze within the zoom reticle. The location may be determined using one or more selection criteria, such as when a user is viewing a particular location of the magnified content for a threshold amount of time. When the location of the user's gaze meets the selection criteria, the location may be used to select a target input. For example, if the location corresponds to an object in a scene, the object may be selected as the target input.

[0120] In some variations, an indicator may be displayed within the zoom reticle. The indicator, which may be of any particular size or shape, may be moved within the zoom reticle to follow the user's gaze. In these cases, the user may move the indicator (e.g., by viewing the zoom reticle) until it is positioned at a desired location within the zoom reticle. The user may perform one or more additional actions (e.g., maintaining their gaze at the desired location for a threshold amount of time, giving a voice command, providing manual input to the electronic device, etc.) to confirm that the indicator is positioned at the desired location. Thus, the location may be used to select a target input as described above.

[0121] In other variations, the electronic device may use a user gesture to select a target. For example, a user may point to a specific location within a zoom reticle, and the location may be used to select a target input. In other variations, the user may move the electronic device to select a target input. Specifically, the movement of the electronic device may be monitored to determine the target input. For example, the user may move the electronic device so that the target input (e.g., presented in the magnified content) is positioned at a specific location in the zoom reticle (e.g., centered in the zoom reticle). For example, an indicator may be displayed in the zoom reticle, and the user may move the electronic device until the indicator is positioned above the location of interest. The user may remain still (or move at a sufficiently slow speed) for a threshold period of time, at which point the electronic device may determine that the user has positioned the zoom reticle relative to the location of interest, and the location of interest may be used to select the target input.

[0122] When the target input is determined, the request may be performed at block 1008 using the target input. In some variations, the zoom reticle may be deactivated after the target input is determined because the zoom reticle may no longer be needed or desired to be present. In some cases, the zoom reticle may be deactivated in response to determining the target input only if the zoom reticle was initially activated in response to receiving the request. Thus, if the zoom reticle was already displayed when the request was received, the zoom reticle may continue to be displayed after the target input is determined. In these cases, the zoom reticle may be deactivated in other circumstances (e.g., in response to a user command to deactivate the zoom reticle).

[0123] Figure 1100 shows the Figure 3A and Figure 3B1000 in the context of a scene 300 of the electronic device 100. Specifically, the depicted portion of the scene 300 represents a boundary of a camera field of view 200 of the electronic device 100. A zoom reticle 306 is displayed in the extended reality environment, and the zoom reticle includes magnified content 308, such as previously described. In response to receiving a request for a target input, the electronic device 100 may determine a location 1100, and the determined location 1100 may be used to select the target input (e.g., as an object corresponding to the determined location 1100). In some cases, the user's gaze may be used to determine the location 1100 (e.g., if the user views the location 1100 for a threshold amount of time, if the user's gaze is used to move the indicator 1102 to the location 1100, etc.). In other cases, the location 1100 may be determined using a user gesture (e.g., the user points to the location 1100). In other variations, the user may move the electronic device 100 so that the location 1100 is positioned within a particular area in the zoom reticle 306 (e.g., positioned within the Fig.11 Once a target input is selected, the electronic device 100 may use the selected target input to execute the request.

[0124] In some variations of the devices and methods described herein, a user gesture may be used to change one or more aspects of the zoom reticle. For example, in some variations, while the electronic device 100 is actively displaying the zoom reticle, the electronic device 100 may be configured to detect the performance of a user gesture (or a portion thereof). In response to detecting a user gesture while the electronic device 100 is displaying the zoom reticle, the electronic device 100 may change one or more aspects of the zoom reticle.

[0125] In some variations, in response to detecting a user gesture, the electronic device 100 may change the magnification level associated with the zoom reticle. In one example, one or more pinch gestures may be used to change the magnification level of the zoom reticle. For example, a pinch gesture (e.g., in which the user spreads two fingers apart) may be used to increase the magnification level of the zoom reticle. When the electronic device 100 detects that the user has performed a pinch gesture, the electronic device 100 may increase the magnification level of the zoom reticle. In some of these variations, each time the user performs a pinch gesture, the electronic device 100 may increase the magnification level by a predetermined amount (e.g., to a maximum magnification level). In other variations, the electronic device 100 may increase the magnification level by an amount that depends on the magnitude of the pinch gesture. In other words, the amount of increase in the magnification level may depend on the extent to which the user spreads their fingers apart while performing the pinch gesture.

[0126] Conversely, a pinch gesture (e.g., in which a user brings two fingers closer together) may be used to reduce the magnification level of the zoom reticle. When electronic device 100 detects that the user has performed a pinch gesture, electronic device 100 may reduce the magnification level of the zoom reticle. In some of these variations, each time the user performs a pinch gesture, electronic device 100 may reduce the magnification level by a predetermined amount (e.g., to a minimum magnification level). In other variations, electronic device 100 may reduce the magnification level by an amount that depends on the magnitude of the pinch gesture. In these cases, the amount of reduction in the magnification level may depend on how closely the user brings their fingers into contact.

[0127] Additionally or alternatively, the electronic device may change the size of the zoom reticle in response to detecting a user gesture. In some variations, different user gestures may be used to change different aspects of the zoom reticle. For example, in response to electronic device 100 detecting that the user has performed a first gesture, electronic device 100 may change the magnification level associated with the zoom reticle, and in response to electronic device 100 detecting that the user has performed a second gesture, electronic device 100 may change the size of the zoom reticle.

[0128] In some variations, different instances of the same user gesture may be used to activate the zoom reticle and change an aspect of the zoom reticle. For example, electronic device 100 may utilize different instances of a first gesture (e.g., a pinch gesture) to activate the zoom reticle and change an aspect of the zoom reticle. If electronic device 100 detects that the user has performed an instance of the first gesture while the zoom reticle is not displayed, electronic device 100 may activate the zoom reticle. Conversely, if electronic device 100 detects that the user has performed an instance of the first gesture while the zoom reticle is displayed, electronic device 100 may change an aspect of the zoom reticle (e.g., increase the magnification level associated with the zoom reticle, increase the size of the zoom reticle, etc.).

[0129] Additionally or alternatively, the same instance of a user gesture may be used to activate the zoom reticle and change an aspect of the zoom reticle. In these cases, the electronic device 100 may activate the zoom reticle in response to detecting a first portion of the user gesture, and may change an aspect of the zoom reticle in response to detecting a second portion of the user gesture. Using a pinch gesture as an example, the electronic device 100 may be configured to activate the zoom reticle in response to determining that the user has moved their fingers apart a threshold amount. If the user continues to move their fingers apart beyond the threshold amount (e.g., as part of the same pinch gesture), the electronic device 100 may change an aspect of the zoom reticle (e.g., increase the magnification level of the zoom reticle).

[0130] In some variations, different instances of the same user gesture may be used to change an aspect of the zoom reticle and deactivate the zoom reticle. For example, electronic device 100 may utilize different instances of a second gesture (e.g., a pinch gesture) to change an aspect of the zoom reticle and deactivate the zoom reticle. For example, if electronic device 100 detects that the user has performed an instance of the second gesture while the zoom reticle is being displayed, electronic device 100 may change an aspect of the zoom reticle (e.g., reduce the magnification level associated with the zoom reticle, reduce the size of the zoom reticle, etc.). If the aspect of the zoom reticle reaches a predetermined state (e.g., a minimum magnification level associated with the zoom reticle, a minimum size of the zoom reticle) and electronic device 100 detects additional instances of the second gesture, electronic device 100 may deactivate the zoom reticle.

[0131] For the purpose of illustration, the foregoing description uses specific nomenclature to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that specific details are not required in order to practice the embodiments. Therefore, for the purpose of illustration and description, the foregoing description of the specific embodiments described herein is presented. They are not intended to be exhaustive or to limit the embodiments to the disclosed precise forms. It will be apparent to those of ordinary skill in the art that, in view of the above teachings, many modifications and variations are feasible.

[0132] According to the disclosure of the present invention, the following embodiments of content-based presentation of a magnified view in an extended reality environment are provided:

[0133] 1. A method for operating an electronic device, the method comprising:

[0134] Selecting a display position of a display of the electronic device;

[0135] selecting an area of ​​a field of view of a camera of the electronic device;

[0136] detecting text in the region;

[0137] In response to detecting the text, selecting an image modification technique; and

[0138] A zoom reticle is displayed at the display location, wherein the zoom reticle includes an enlarged portion of the region modified according to the selected image modification technique.

[0139] 2. The method according to claim 1, wherein displaying the zoom reticle comprises:

[0140] The magnified portion of the area is displayed in a first sub-area of ​​the zoom reticle.

[0141] 3. The method according to item 2, wherein displaying the zoom reticle comprises:

[0142] An additional magnified portion of the region is displayed in a second sub-region of the zoom reticle, wherein:

[0143] The additional enlarged portion of the region is not modified according to the selected image modification technique.

[0144] 4. The method according to any one of items 1 to 3, wherein:

[0145] The image modification techniques include contrast adjustment.

[0146] 5. The method according to any one of items 1 to 4, wherein:

[0147] The image modification techniques include brightness adjustment.

[0148] 6. The method according to any one of items 1 to 3, wherein:

[0149] The image modification technique includes replacing image data from the region with synthetic image content.

[0150] 7. The method according to any one of items 1 to 6, comprising:

[0151] determining a character size of the detected text; and

[0152] Use the determined character size to select the magnification level, where:

[0153] The enlarged portion of the region is enlarged according to the enlargement level.

[0154] 8. The method according to item 7, wherein:

[0155] Selecting the magnification level includes using a target character size to select the magnification level.

[0156] 9. The method according to any one of items 1 to 8, comprising:

[0157] receiving a request to activate the zoom reticle; and

[0158] In response to receiving the request, the zoom reticle is displayed at the display position.

[0159] 10. A non-transitory computer-readable medium comprising instructions which, when executed by at least one computing device, cause the at least one computing device to perform operations comprising the steps of any one of items 1 to 9.

[0160] 11. An electronic device, comprising:

[0161] a camera having a field of view;

[0162] Display; and

[0163] a processor operably coupled to the camera and the display, the processor configured to:

[0164] Selecting a display position of the display;

[0165] selecting an area of ​​a field of view of the camera of the electronic device;

[0166] detecting text in the region;

[0167] In response to detecting the text, selecting an image modification technique; and

[0168] displaying a zoom graticule at the display position, wherein the zoom graticule includes an enlarged portion of the region modified according to the selected image modification technique. 12. The electronic device of item 11, wherein displaying the zoom graticule comprises:

[0169] The magnified portion of the area is displayed in a first sub-area of ​​the zoom reticle.

[0170] 13. The electronic device according to item 12, wherein displaying the zoom reticle comprises:

[0171] Display an additional magnified portion of the area in a second sub-area of ​​the zoom reticle

[0172] points, of which:

[0173] The additional enlarged portion of the region is not modified according to the selected image modification technique.

[0174] 14. An electronic device according to any one of items 11 to 13, wherein:

[0175] The image modification techniques include contrast adjustment.

[0176] 15. An electronic device according to any one of items 11 to 14, wherein:

[0177] The image modification techniques include brightness adjustment.

[0178] 16. An electronic device according to any one of items 11 to 13, wherein:

[0179] The image modification technique includes replacing image data from the region with synthetic image content.

[0180] 17. The electronic device according to any one of items 11 to 16, wherein the processor is configured to:

[0181] determining a character size of the detected text; and

[0182] Use the determined character size to select the magnification level, where:

[0183] The enlarged portion of the region is enlarged according to the enlargement level.

[0184] 18. The electronic device according to item 17, wherein:

[0185] Selecting the magnification level includes using a target character size to select the magnification level.

[0186] 19. The electronic device according to any one of items 11 to 18, wherein the processor is configured to:

[0187] receiving a request to activate the zoom reticle; and

[0188] In response to receiving the request, the zoom reticle is displayed at the display position.

[0189] 20. The electronic device of any one of items 11 to 19, wherein detecting text comprises:

[0190] Optical character recognition is performed on the area of ​​the field of view of the camera.

[0191] 21. A method of operating an electronic device, the method comprising:

[0192] receiving a request to activate a zoom reticle; and

[0193] In response to the request:

[0194] Selecting a display position of a display of the electronic device;

[0195] selecting an area of ​​a field of view of the camera of the electronic device;

[0196] determining a content type associated with the region;

[0197] selecting a set of display attributes based on the determined content type; and

[0198] and displaying a zoom graticule at the display location, wherein the zoom graticule includes an enlarged portion of the area displayed according to a selected set of display attributes.

[0199] The content type is selected from a list of candidate content types including text content types and non-text content types.

[0200] 23. The method according to item 21 or item 22, wherein:

[0201] The set of display properties includes a magnification level.

[0202] 24. The method according to any one of items 21 to 23, wherein:

[0203] The set of display properties includes a contrast setting.

[0204] 25. The method according to any one of items 21 to 24, wherein:

[0205] The set of display properties includes a brightness setting.

[0206] 26. The method of any one of items 21 to 25, wherein displaying the zoom reticle comprises:

[0207] The enlarged portion of the area is displayed in a first area of ​​the zoom reticle.

[0208] 27. The method according to item 26, comprising:

[0209] The first region is selected based on the determined content type.

[0210] 28. The method of item 26 or item 27, wherein displaying the zoom reticle comprises:

[0211] An additional magnified portion of the region is displayed in a second region of the zoom reticle, wherein:

[0212] The additional enlarged portion of the region is displayed according to a different set of display attributes.

[0213] 29. A non-transitory computer-readable medium comprising instructions which, when executed by at least one computing device, cause the at least one computing device to perform operations comprising the steps of any one of items 21 to 28.

[0214] 30. An electronic device, comprising:

[0215] a camera having a field of view;

[0216] Display; and

[0217] a processor operably coupled to the camera and the display, the processor configured to:

[0218] selecting a display position of the display of the electronic device;

[0219] selecting an area of ​​a field of view of the camera of the electronic device;

[0220] determining a content type associated with the region;

[0221] selecting a set of display attributes based on the determined content type; and

[0222] and displaying a zoom graticule at the display location, wherein the zoom graticule includes an enlarged portion of the area displayed according to the selected set of display attributes. 31. The electronic device of item 30, wherein determining the content type comprises:

[0223] The content type is selected from a list of candidate content types including text content types and non-text content types.

[0224] 32. The electronic device according to item 30 or 31, wherein:

[0225] The set of display properties includes a magnification setting.

[0226] 33. An electronic device according to any one of items 30 to 32, wherein:

[0227] The set of display properties includes a contrast setting.

[0228] 34. An electronic device according to any one of items 30 to 33, wherein:

[0229] The set of display properties includes a brightness setting.

[0230] 35. The electronic device according to any one of items 30 to 34, wherein displaying the zoom reticle comprises:

[0231] The enlarged portion of the area is displayed in a first area of ​​the zoom reticle.

[0232] 36. The electronic device according to item 35, wherein the processor is configured to:

[0233] The first region is selected based on the determined content type.

[0234] 37. The electronic device according to item 35 or item 36, wherein displaying the zoom reticle comprises:

[0235] An additional magnified portion of the region is displayed in a second region of the zoom reticle, wherein:

[0236] The additional enlarged portion of the region is displayed according to a different set of display attributes.

[0237] 38. A method of operating an electronic device, the method comprising:

[0238] receiving a request to activate a zoom reticle; and

[0239] In response to the request:

[0240] Selecting a display position of a display of the electronic device;

[0241] detecting text in an area of ​​a field of view of the camera of the electronic device;

[0242] selecting a magnification level based on the detected text; and

[0243] The zoom reticle is displayed at the display position, wherein the zoom reticle includes a portion of the area magnified according to the magnification level.

[0244] 39. The method according to item 38, comprising:

[0245] A character size of the detected text is determined, wherein the magnification level is selected using the determined character size.

[0246] 40. The method according to item 38 or item 39, wherein:

[0247] Selecting the magnification level includes using a target character size to select the magnification level.

[0248] In addition, according to the disclosure of the present invention, the following embodiments of target input selection using a magnified view in an extended reality environment are provided:

[0249] 1. A method for operating an electronic device, the method comprising:

[0250] receiving a request for target input in a field of view of a camera of the electronic device;

[0251] displaying a zoom reticle at a display of the electronic device, wherein the zoom reticle comprises a magnified portion of the field of view of the camera of the electronic device;

[0252] determining the target input using the magnified portion of the field of view; and

[0253] The request is executed relative to the target input.

[0254] 2. The method according to item 1, comprising:

[0255] The zoom reticle is activated in response to receiving the request.

[0256] 3. The method according to item 1, wherein:

[0257] The request is received while the zoom reticle is displayed.

[0258] 4. The method according to any one of items 1 to 3, wherein:

[0259] The request is a voice command from the user.

[0260] 5. The method according to any one of items 1 to 4, wherein determining the target input comprises:

[0261] detecting user interaction with the zoom reticle; and

[0262] The target input is selected based on a detected user interaction with the zoom reticle.

[0263] 6. The method according to item 6, wherein:

[0264] Detecting the user interaction with the zoom reticle includes determining a location of a user's gaze in the zoom reticle; and

[0265] The target input is selected using the determined position.

[0266] 7. The method according to item 5, wherein:

[0267] Detecting the user interaction with the zoom reticle includes detecting movement of the electronic device.

[0268] 8. The method according to item 5, wherein:

[0269] Detecting the user interaction with the zoom reticle includes detecting a user gesture corresponding to the zoom reticle.

[0270] 9. The method according to any one of items 1 to 8, comprising:

[0271] The zoom reticle is deactivated after determining the target input.

[0272] 10. A non-transitory computer-readable medium comprising instructions which, when executed by at least one computing device, cause the at least one computing device to perform operations comprising the steps of any one of items 1 to 9.

[0273] 11. An electronic device, comprising:

[0274] a camera having a field of view;

[0275] Display; and

[0276] a processor operably coupled to the camera and the display, the processor configured to:

[0277] receiving a request for target input in a field of view of the camera of the electronic device;

[0278] displaying a zoom reticle at the display of the electronic device, wherein the zoom reticle comprises a magnified portion of the field of view of the camera of the electronic device;

[0279] determining the target input using the magnified portion of the field of view; and

[0280] The request is executed relative to the target input.

[0281] 12. The electronic device according to item 11, wherein the processor is configured to:

[0282] The zoom reticle is activated in response to receiving the request.

[0283] 13. The electronic device according to item 11, wherein:

[0284] The request is received while the zoom reticle is displayed.

[0285] 14. An electronic device according to any one of items 11 to 13, wherein:

[0286] The request is a voice command from the user.

[0287] 15. The electronic device according to any one of items 11 to 14, wherein determining the target input comprises:

[0288] detecting user interaction with the zoom reticle; and

[0289] The target input is selected based on a detected user interaction with the zoom reticle.

[0290] 16. The electronic device according to item 15, wherein:

[0291] Detecting the user interaction with the zoom reticle includes determining a location of a user's gaze in the zoom reticle; and

[0292] The target input is selected using the determined position.

[0293] 17. The electronic device according to item 16, comprising:

[0294] An eye tracker is configured to detect the gaze of the user.

[0295] 18. The electronic device according to item 15, wherein:

[0296] Detecting the user interaction with the zoom reticle includes measuring movement of the electronic device.

[0297] 19. The electronic device according to item 15, wherein:

[0298] Detecting the user interaction with the zoom reticle includes detecting a user gesture corresponding to the zoom reticle.

[0299] 20. The electronic device according to any one of items 11 to 19, wherein the processor is configured to:

[0300] The zoom reticle is deactivated after determining the target input.

Claims

1. A method of operating an electronic device, comprising: During a first time period, collecting gaze position information of the user within a first area of ​​the gaze field; During a second time period after the first time period, detecting movement of the gaze of the user to a second area of ​​the gaze field of view; In response to detecting the movement of the gaze of the user to the second area, selecting a display position of a display of the electronic device, wherein the display position is selected using the position information collected during the first time period; as well as A zoom graticule is displayed at the display position, wherein the zoom graticule includes an enlarged portion of the gaze field.

2. The method of claim 1, wherein selecting the display position comprises: detecting a target object in the gaze field using the position information collected during the first time period; as well as The display position is selected using the position of the target object in the gaze field.

3. The method according to claim 2, comprising: The movement of the target object in the gaze field is detected.

4. The method according to claim 3, comprising: using the detected motion of the target object to select an updated display position of the display; as well as The zoom reticle is moved to the updated display position.

5. The method according to claim 3 or claim 4, comprising: The detected motion of the target object is used to change the magnification level of the zoom reticle.

6. The method according to any one of claims 1 to 4, wherein: The display has a display area positioned to partially overlap the gaze field; and The display position is selected so that the zoom reticle is positioned within the display area.

7. The method according to claim 6, wherein: The second area of ​​the gaze field is positioned outside the display area.

8. The method of claim 6, wherein selecting the display position comprises: detecting a target object in the gaze field using the position information detected during the first time period, wherein the target object is located outside the display area; as well as The display position is selected using the position of the target object in the gaze field.

9. An electronic device, comprising: an eye tracker configured to detect a gaze of a user within a gaze field of view; monitor; a processor operably coupled to the eye tracker and the display, the processor configured to: During a first time period, collecting gaze position information of the user in a first area of ​​the gaze field of view; During a second time period after the first time period, detecting movement of the user's gaze to a second area of ​​the gaze field of view; In response to detecting the movement of the gaze of the user to the second area, selecting a display position of the display of the electronic device, wherein the display position is selected using the position information collected during the first time period; as well as A zoom graticule is displayed at the display position, wherein the zoom graticule includes an enlarged portion of the gaze field.

10. The electronic device according to claim 9, comprising: A camera having an imaging field of view that at least partially overlaps the gaze field of view.

11. The electronic device according to claim 9 or claim 10, wherein selecting the display position comprises: detecting a target object in the gaze field using the position information detected during the first time period; as well as The display position is selected using the position of the target object in the gaze field.

12. The electronic device of claim 11, wherein the processor is configured to: The movement of the target object in the gaze field is detected.

13. The electronic device of claim 12, wherein the processor is configured to: using the detected motion of the target object to select an updated display position for the display; and The zoom reticle is moved to the updated display position.

14. The electronic device according to claim 12 or claim 13, wherein the processor is configured to: The detected motion of the target object is used to change the magnification level of the zoom reticle.

15. The electronic device according to claim 9, wherein: The display has a display area positioned to partially overlap the gaze field; and The display position is selected so that the zoom reticle is positioned within the display area.

16. The electronic device according to claim 15, wherein: The second area of ​​the gaze field is positioned outside the display area.

17. The electronic device according to claim 15 or claim 16, wherein selecting the display position comprises: detecting a target object in the gaze field using the position information collected during the first time period, wherein the target object is located outside of the display area; as well as The display position is selected using the position of the target object in the gaze field.

18. A method of operating an electronic device, comprising: During a first time period, collecting gaze position information of the user within the gaze field; receiving a request to activate a zoom reticle, the request having a request type; as well as In response to receiving the request: selecting an analysis technique based on the request type; selecting a display position of a display of the electronic device, wherein the display position is selected based on an analysis of the collected information using the selected analysis technique; as well as A zoom graticule is displayed at the selected display position, wherein the zoom graticule includes an enlarged portion of the gaze field.

19. The method of claim 18, wherein: The request type is selected from a list of candidate request types including voice command types.

20. The method of claim 19, wherein the list of candidate request types includes an attention-based request type.

Images