Suggest content based on physical objects in the environment

Abstract

The subject matter of the present disclosure is "Suggesting content based on physical objects in an environment". Some examples of the present disclosure relate to systems and methods for presenting one or more user interface elements including suggested content based on detecting physical objects in a computer-generated environment. In some examples, the electronic device captures one or more images of a physical environment including a first object (e.g., a fillable form). In some examples, based on detecting a first area (e.g., a first fillable field) of the first object in the one or more images and satisfying one or more first criteria, the electronic device displays a first user interface element including first suggested text, wherein the first user interface element is displayed at a first position based on the first area.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 586,969 filed on September 29, 2023, U.S. Provisional Application No. 63 / 556,592 filed on February 22, 2024, and U.S. Patent Application No. 18 / 892,264 filed on September 20, 2024, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present invention generally relates to systems and methods for presenting user interface elements including suggested text based on detecting one or more regions of a physical object in a three-dimensional environment. Background Art

[0004] Some computer graphics environments provide two-dimensional and / or three-dimensional environments in which at least some of the objects displayed for user viewing are virtual and generated by a computer. In some examples, a physical environment (e.g., including one or more physical objects) is optionally presented in a three-dimensional environment along with one or more virtual objects. Summary of the invention

[0005] Some examples of the present disclosure relate to systems and methods for presenting one or more user interface elements including suggested content based on detecting a physical object in a computer-generated environment. In some examples, a method is performed at an electronic device that communicates with one or more displays, one or more input devices, and one or more cameras. In some examples, the electronic device captures one or more images of a physical environment including a first object (e.g., a form with fillable fields) via the one or more cameras. In some examples, based on detecting a first area (e.g., a first fillable field) of the first object in the one or more images and satisfying one or more first criteria, the electronic device displays a first user interface element including a first suggested text in a computer-generated environment via the one or more displays, wherein the first user interface element is displayed at a first position based on the first area. In some examples, based on detecting a second area (e.g., a second fillable field) of the first object different from the first area in the one or more images and satisfying one or more second criteria, the electronic device displays a second user interface element including a second suggested text in a computer-generated environment via the one or more displays, wherein the second user interface element is displayed at a second position different from the first position based on the second area.

[0006] In some examples, satisfying the one or more first criteria and the one or more second criteria is based on whether a user intent to interact with the first object is detected via the one or more input devices when the first area or the second area of ​​the first object is detected. In some examples, when the first area of ​​the first object is detected, the one or more first criteria include a criterion satisfied when the electronic device detects the gaze of the user pointing to the first area, and when the second area of ​​the first object is detected, the one or more second criteria include a criterion satisfied when the electronic device detects the gaze of the user pointing to the second area. In some examples, when the first area of ​​the first object is detected, the one or more first criteria include a criterion satisfied when the electronic device detects a gesture performed by a user's finger or hand pointing to the first area (e.g., using a finger to point to a fillable field), and when the second area of ​​the first object is detected, the one or more second criteria include a criterion satisfied when the electronic device detects a gesture performed by the user's hand pointing to the second area. In some examples, when the first area of ​​the first object is detected, the one or more first criteria include a criterion satisfied when the electronic device detects a second object associated with the user's hand pointing to the first area, and when the second area of ​​the first object is detected, the one or more second criteria include a criterion satisfied when the electronic device detects a second object associated with the user's hand pointing to the second area.

[0007] A comprehensive description of these examples is provided in the drawings and detailed description, and it should be understood that this summary does not limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a better understanding of the various examples described herein, reference should be made to the following detailed description and the following drawings. Throughout the drawings, like reference numerals generally refer to corresponding parts.

[0009] Figure 1 An electronic device presenting an extended reality environment according to some examples of the present disclosure is shown.

[0010] Figure 2 A block diagram illustrating an example architecture of a device according to some examples of the present disclosure.

[0011] FIG. 3A to FIG. 3M An example is shown in which an electronic device according to some examples of the present disclosure displays a user interface element including suggested text based on detecting one or more areas of a physical object.

[0012] FIG. 4A to FIG. 4C An example is shown in which an electronic device according to some examples of the present disclosure displays a user interface element including suggested text based on detecting one or more areas of a physical object.

[0013] Figure 5 is a flow chart illustrating an exemplary process for displaying a user interface element including suggested text based on detecting an area of ​​a physical object according to some examples of the present disclosure. DETAILED DESCRIPTION

[0014] Some examples of the present disclosure relate to systems and methods for presenting one or more user interface elements including suggested content based on detecting a physical object in a computer-generated environment. In some examples, a method is performed at an electronic device that communicates with one or more displays, one or more input devices, and one or more cameras. In some examples, the electronic device captures one or more images of a physical environment including a first object (e.g., a form with fillable fields) via the one or more cameras. In some examples, based on detecting a first area (e.g., a first fillable field) of the first object in the one or more images and satisfying one or more first criteria, the electronic device displays a first user interface element including a first suggested text in a computer-generated environment via the one or more displays, wherein the first user interface element is displayed at a first position based on the first area. In some examples, based on detecting a second area (e.g., a second fillable field) of the first object different from the first area in the one or more images and satisfying one or more second criteria, the electronic device displays a second user interface element including a second suggested text in a computer-generated environment via the one or more displays, wherein the second user interface element is displayed at a second position different from the first position based on the second area.

[0015] In some examples, satisfying the one or more first criteria and the one or more second criteria is based on whether a user intent to interact with the first object is detected via the one or more input devices when the first area or the second area of ​​the first object is detected. In some examples, when the first area of ​​the first object is detected, the one or more first criteria include a criterion satisfied when the electronic device detects the gaze of the user pointing to the first area, and when the second area of ​​the first object is detected, the one or more second criteria include a criterion satisfied when the electronic device detects the gaze of the user pointing to the second area. In some examples, when the first area of ​​the first object is detected, the one or more first criteria include a criterion satisfied when the electronic device detects a gesture performed by a user's finger or hand pointing to the first area (e.g., using a finger to point to a fillable field), and when the second area of ​​the first object is detected, the one or more second criteria include a criterion satisfied when the electronic device detects a gesture performed by the user's hand pointing to the second area. In some examples, when the first area of ​​the first object is detected, the one or more first criteria include a criterion satisfied when the electronic device detects a second object associated with the user's hand pointing to the first area, and when the second area of ​​the first object is detected, the one or more second criteria include a criterion satisfied when the electronic device detects a second object associated with the user's hand pointing to the second area.

[0016] In some examples, a three-dimensional object is displayed in a computer-generated three-dimensional environment at a particular orientation that controls one or more behaviors of the three-dimensional object (e.g., when the three-dimensional object moves within the three-dimensional environment). In some examples, the orientation in which the three-dimensional object is displayed in the three-dimensional environment is selected by a user of the electronic device or automatically selected by the electronic device. For example, when initiating presentation of the three-dimensional object in the three-dimensional environment, the user may select a particular orientation of the three-dimensional object, or the electronic device may automatically select the orientation of the three-dimensional object (e.g., based on the type of the three-dimensional object).

[0017] In some examples, a three-dimensional object may be displayed in a three-dimensional environment in a world-locked orientation, a body-locked orientation, a tilt-locked orientation, or a head-locked orientation, as described below. As used herein, an object displayed in a three-dimensional environment in a body-locked orientation has a distance and orientation offset relative to a portion of the user's body (e.g., the user's torso). Alternatively, in some examples, the body-locked object has a fixed distance from the user, and the orientation of the content does not reference any part of the user's body (e.g., may be displayed in the same basic direction relative to the user regardless of head and / or body movement). Additionally or alternatively, in some examples, the body-locked object may be configured to always maintain gravity or horizon (e.g., perpendicular to gravity) alignment, so that changes in the head and / or body in the rolling direction will not cause the body-locked object to move within the three-dimensional environment. Instead, translational movement in either configuration will cause the body-locked object to reposition within the three-dimensional environment to maintain the distance offset.

[0018] As used herein, an object displayed in a head-locked orientation in a three-dimensional environment has a distance and orientation offset relative to the user's head. In some examples, the head-locked object moves within the three-dimensional environment when the user's head moves (when the user's viewpoint changes).

[0019] As used herein, an object displayed in a world-locked orientation in a three-dimensional environment has no distance or orientation offset relative to the user.

[0020] As used herein, an object displayed in a three-dimensional environment with a tilt-locked orientation (referred to herein as a tilt-locked object) has a distance offset relative to a user, such as a portion of a user's body (e.g., a user's torso) or a user's head. In some examples, the tilt-locked object is displayed with a fixed orientation relative to the three-dimensional environment. In some examples, the tilt-locked object moves according to a polar (e.g., spherical) coordinate system centered on a pole passing through the user (e.g., the user's head). For example, the tilt-locked object is moved in the three-dimensional environment based on the movement of the user's head in a spherical space around the user's head (e.g., centered on the user's head). Thus, if the user tilts his head relative to gravity (e.g., up or down in a pitch direction), the tilt-locked object will follow the head tilt and move radially along the sphere, so that the tilt-locked object is repositioned within the three-dimensional environment to the same distance offset relative to the user as before the head tilt, while optionally maintaining the same orientation relative to the three-dimensional environment. In some examples, if the user moves his head in a rolling direction (e.g., clockwise or counterclockwise) relative to gravity, the tilt-locked object is not repositioned within the three-dimensional environment.

[0021] Figure 1An electronic device 101 is illustrated that presents an extended reality (XR) environment (e.g., a computer-generated environment that optionally includes representations of physical and / or virtual objects) according to some examples of the present disclosure. In some examples, such as Figure 1 As shown, the electronic device 101 is a head mounted display or other head mounted device configured to be worn on the head of the user of the electronic device 101. Figure 2 An example of an electronic device 101 is described by using an architectural block diagram of FIG. Figure 1 As shown, electronic device 101 and table 106 are located in a physical environment. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lights, etc.). In some examples, electronic device 101 may be configured to detect and / or capture an image of the physical environment including table 106 (shown in the field of view of electronic device 101).

[0022] In some examples, such as Figure 1 As shown, the electronic device 101 includes one or more internal image sensors 114a (eg, as described below with reference to FIG. 1 ) oriented toward the user's face. Figure 2 101 and 102. In some examples, the internal image sensor 114a is used for eye tracking (e.g., detecting the gaze of the user). The internal image sensor 114a is optionally arranged on the left and right portions of the display 120 to enable eye tracking of the user's left and right eyes. In some examples, the electronic device 101 also includes external image sensors 114b and 114c facing outward from the user to detect and / or capture the physical environment of the electronic device 101 and / or the movement of the user's hands or other body parts.

[0023] In some examples, the display 120 has a field of view visible to the user (e.g., it may or may not correspond to the field of view of the external image sensors 114b and 114c). Because the display 120 is optionally part of the head-mounted device, the field of view of the display 120 is optionally the same as or similar to the field of view of the user's eyes. In other examples, the field of view of the display 120 may be smaller than the field of view of the user's eyes. In some examples, the electronic device 101 may be an optical see-through device, in which the display 120 is a transparent or translucent display through which parts of the physical environment can be directly viewed. In some examples, the display 120 may be included in a transparent lens and may overlap with all or only a portion of the transparent lens. In other examples, the electronic device may be a video pass-through device, in which the display 120 is an opaque display configured to display an image of the physical environment captured by the external image sensors 114b and 114c.

[0024] In some examples, in response to a trigger, the electronic device 101 may be configured to display in the XR environment Figure 1 , which does not exist in the physical environment but is displayed in the XR environment on top of the real-world table 106 (or its representation). Optionally, in response to detecting the plane of the table 106 in the physical environment 100, the virtual object 104 can be displayed on the surface of the table 106 in the XR environment displayed via the display 120 of the electronic device 101.

[0025] It should be understood that virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional XR environment. For example, a virtual object may represent an application or user interface displayed in an XR environment. In some examples, a virtual object may represent content corresponding to an application and / or displayed via a user interface in an XR environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures, such as air pinch gestures, air tap gestures, and / or air touch gestures), so that a user virtually touches, taps, moves, rotates, or otherwise interacts with virtual object 104.

[0026] In some examples, displaying an object in a three-dimensional environment may include interaction with one or more user interface objects in the three-dimensional environment. For example, initiation of display of an object in a three-dimensional environment may include interaction with one or more virtual options / indicators displayed in the three-dimensional environment. In some examples, when display of an object is initiated in a three-dimensional environment, an electronic device may track the user's gaze as an input for identifying one or more virtual options / indicators as targets for selection. For example, gaze may be used to identify one or more virtual options / indicators as targets for selection using another selection input. In some examples, a virtual option / indicator may be selected using hand tracking input detected via an input device that communicates with the electronic device. In some examples, an object displayed in a three-dimensional environment may be moved and / or reoriented in the three-dimensional environment based on movement input detected via an input device.

[0027] In the following discussion, an electronic device that communicates with a display generation component and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in the present disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device to be displayed on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as used in the present disclosure, input received on an electronic device (e.g., touch input received on a touch-sensitive surface of an electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, and the electronic device receives input information from the independent input device.

[0028] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and / or a digital video player application.

[0029] Figure 2 A block diagram of an example architecture of an electronic device 201 according to some examples of the present disclosure is illustrated. In some examples, the electronic device 201 includes one or more electronic devices. For example, the electronic device 201 can be a portable device, an auxiliary device that communicates with another device, a head-mounted display, etc. In some examples, the electronic device 201 corresponds to the above reference Figure 1 An electronic device 101 is described.

[0030] like Figure 2 As illustrated, the electronic device 201 optionally includes various sensors, such as one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206 (optionally corresponding to Figure 1114a and / or external image sensors 114b and 114c in the embodiment of the present invention), one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors), one or more display generation components 214 (optionally corresponding to Figure 1 The display 120 in the electronic device 201, one or more speakers 216, one or more processors 218, one or more memories 220 and / or communication circuits 222. One or more communication buses 208 are optionally used for communication between the above components of the electronic device 201.

[0031] The communication circuit 222 optionally includes circuits for communicating with electronic devices, networks such as the Internet, intranets, wired networks and / or wireless networks, cellular networks, and wireless local area networks (LANs). The communication circuit 222 optionally includes circuits for using near field communication (NFC) and / or short-range communication such as A circuit for communicating.

[0032] Processor 218 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 220 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processor 218 to perform the techniques, processes, and / or methods described below. In some examples, memory 220 may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in combination with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, a magnetic storage device, an optical storage device, and / or a semiconductor storage device. Examples of such storage devices include magnetic disks, optical disks based on compact disc (CD), digital versatile disc (DVD) or Blu-ray technology, and persistent solid-state memory (such as flash memory, solid-state drives, etc.).

[0033] In some examples, display generation component 214 includes a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation component 214 includes multiple displays. In some examples, display generation component 214 may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, electronic device 201 includes touch-sensitive surfaces 209 for receiving user inputs (such as tap inputs and swipe inputs or other gestures). In some examples, display generation component 214 and touch-sensitive surface 209 form a touch-sensitive display (e.g., a touch screen integrated with electronic device 201 or a touch screen that communicates with electronic device 201 outside electronic device 201).

[0034] The electronic device 201 optionally includes an image sensor 206. The image sensor 206 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real-world environment. The image sensor 206 also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensor 206 also optionally includes one or more cameras configured to capture the movement of physical objects in the real-world environment. The image sensor 206 also optionally includes one or more depth sensors configured to detect the distance of the physical object from the electronic device 201. In some examples, information from the one or more depth sensors may allow the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some examples, the one or more depth sensors may allow the device to determine the texture and / or topography of an object in the real-world environment.

[0035] In some examples, the electronic device 201 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around the electronic device 201. In some examples, the image sensor 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, the electronic device 201 uses the image sensor 206 to detect the positioning and orientation of the electronic device 201 and / or the display generation component 214 in the real-world environment. For example, the electronic device 201 uses the image sensor 206 to track the positioning and orientation of the display generation component 214 relative to one or more fixed objects in the real-world environment.

[0036] In some examples, the electronic device 201 includes a microphone 213 or other audio sensor. The electronic device 201 optionally uses the microphone 213 to detect sounds from the user and / or the user's real-world environment. In some examples, the microphone 213 includes a microphone array (multiple microphones) that optionally operate in conjunction to identify ambient noise or locate a sound source in the space of the real-world environment.

[0037] The electronic device 201 includes a location sensor 204 for detecting the location of the electronic device 201 and / or the display generating component 214. For example, the location sensor 204 may include a global positioning system (GPS) receiver that receives data from one or more satellites and allows the electronic device 201 to determine the absolute location of the device in the physical world.

[0038] The electronic device 201 includes an orientation sensor 210 for detecting the orientation and / or movement of the electronic device 201 and / or the display generation component 214. For example, the electronic device 201 uses the orientation sensor 210 to track changes in the position and / or orientation of the electronic device 201 and / or the display generation component 214, such as relative to physical objects in the real world environment. The orientation sensor 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0039] In some examples, the electronic device 201 includes a hand tracking sensor 202 and / or an eye tracking sensor 212 and / or other body tracking sensors, such as a leg tracking sensor, a torso tracking sensor, and / or a head tracking sensor. The hand tracking sensor 202 is configured to track the position / location of one or more parts of the user's hand, and / or the movement of one or more parts of the user's hand relative to the extended reality environment, relative to the display generation component 214, and / or relative to another defined coordinate system. The eye tracking sensor 212 is configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented together with the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented separately from the display generation component 214.

[0040] In some examples, the hand tracking sensor 202 (and / or other body tracking sensors, such as leg, torso, and / or head tracking sensors) may use an image sensor 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more body parts (e.g., the legs, torso, head, or hands of a human user). In some examples, the hand may be resolved with sufficient resolution to distinguish fingers and their corresponding positioning. In some examples, one or more image sensors 206 are positioned relative to the user to define a field of view and an interaction space of the image sensor 206, in which the finger / hand position, orientation, and / or movement captured by the image sensor is used as input (e.g., to distinguish from the user's resting hand or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0041] In some examples, eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source such as an LED) that emits light toward the user's eyes. The eye tracking camera may be pointed toward the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by corresponding eye tracking cameras and illumination sources, and focus / gaze may be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by one or more corresponding eye tracking cameras / illumination sources.

[0042] The electronic device 201 is not limited to Figure 2Rather than components and configurations of the electronic device 201, it may include fewer components, other components, or additional components in a variety of configurations. In some examples, the electronic device 201 may be implemented between two electronic devices (e.g., as a system). In some such examples, each (or more) electronic device may each include one or more of the same components discussed above, such as various sensors, one or more display generating components, one or more speakers, one or more processors, one or more memories, and / or communication circuits. One or more people using the electronic device 201 are optionally referred to herein as one or more users of the device.

[0043] Attention is now directed to interaction with one or more virtual objects displayed in a three-dimensional environment presented at an electronic device (e.g., corresponding to electronic device 201). Specifically, in some examples, when the physical environment is visible in the three-dimensional environment, the electronic device visually detects one or more areas of a physical object in the physical environment, such as one or more fillable fields of a document (e.g., a physical document or a digital document displayed on a second electronic device). In some examples, in response to visually detecting the one or more areas of the physical object, the electronic device displays one or more user interface elements including suggested text in the three-dimensional environment, wherein the suggested text is generated based on one or more characteristics of the one or more areas of the physical object.

[0044] FIG. 3A to FIG. 3M An example of an electronic device displaying a user interface element including suggested text based on detecting one or more areas of a physical object according to some examples of the present disclosure is shown. Electronic device 301 may be similar to electronic device 101 or 201 discussed above, and / or may be a head-mounted system / device and / or a projection-based system / device (including a hologram-based system / device) configured to generate and present a three-dimensional environment, such as a head-up display (HUD), a head-mounted display (HMD), a window with integrated display capabilities, or a display formed as a lens designed to be placed on a person's eye (e.g., similar to a contact lens). FIG. 3A to FIG. 3M In the example of , the user optionally wears the electronic device 301 so that the three-dimensional environment 350 (e.g., a computer-generated environment) can be defined by X, Y, and Z axes viewed from the perspective of the electronic device (e.g., a viewpoint associated with the user of the electronic device 301). Therefore, as used herein, the electronic device 301 is configured to be able to move with six degrees of freedom based on the movement of the user (e.g., the user's head), so that the electronic device 301 can move in a roll direction, a pitch direction, and / or a yaw direction.

[0045] like Figure 3A As shown, the electronic device 301 may be positioned in a physical environment (eg, an indoor environment) including a plurality of real-world objects. Figure 3A In the example, the electronic device 301 may be positioned on a table 310 (eg, corresponding to a Figure 1 Thus, in some examples, the three-dimensional environment 350 presented using the electronic device 301 optionally includes captured portions of the physical environment 318 surrounding the electronic device 301, such as one or more representations of the table 310 in the field of view of the three-dimensional environment 350. Figure 3A As shown, three-dimensional environment 350 may include representations of book 311, writing tool box 312, and document 315 positioned atop table 310 in physical environment 318 in which electronic device 301 is located. In some examples, these representations may include portions of the physical environment viewed through a transparent or semi-transparent display of electronic device 301.

[0046] As mentioned above and as Figure 3A As shown, the physical environment 318 includes a document 315, which optionally corresponds to a fillable form including a plurality of fillable fields (e.g., a text input field, an image input field, etc.). In some examples, the plurality of fillable fields of the document 315 prompts a user to provide (e.g., write, draw, etc.) information specific to one or more users, such as a user of the electronic device 301 or another user associated with the user of the electronic device 301 (e.g., a family member, a spouse or partner, a child or other family member, a friend, etc.). For example, Figure 3A As shown, document 315 corresponds to a parking permit form and prompts the user to provide information related to parking the vehicle at a specific location (e.g., a place of business, a residence, etc.). Figure 3A As shown, the document 315 optionally includes multiple areas 316 (e.g., fillable fields), including a first area (316-1) indicating the name of the owner / operator of the vehicle (e.g., driver), a second area (316-2) indicating the driver's license number of the person operating the vehicle, a third area (316-3) indicating a contact number (e.g., mobile phone or telephone number) of the person operating the vehicle, a fourth area (316-4) indicating the make and model of the vehicle, and a fifth area (316-5) indicating the license plate number of the vehicle.

[0047] In some cases, in order to reduce the cognitive burden of a user when filling out one or more areas in a corresponding document / form (e.g., document 315), it may be desirable to provide the user with one or more user interface elements including suggested content (e.g., suggested text, images, etc.), which assist the user in filling out the one or more areas in the corresponding document. For example, the electronic device 301 may be configured to present one or more user interface elements (e.g., based on suggested text presented via the one or more user interface elements) in a three-dimensional environment 350 that assist the user in filling out the multiple areas 316 in the document 315, as discussed in more detail below. Providing suggested content that assists the user in filling out a particular form also reduces the user's time consumption, such as by reducing the time it takes for the user to locate and / or identify the information required to fill out the form.

[0048] In some examples, in response to detecting a user's intent to interact with document 315 (e.g., rather than based solely on detecting document 315 in the field of view of electronic device 301), electronic device 301 displays one or more user interface elements including suggested text that assist the user in filling out fields in document 315. For example, Figure 3B As shown, the electronic device 301 detects the user's gaze 321 in the three-dimensional environment 350, which points to the first area 316-1 in the document 315. In some examples, such as Figure 3B As shown, the electronic device 301 detects that the gaze 321 is directed to the first area 316-1 for a threshold amount of time (e.g., 0.5, 1, 1.5, 2, 3, 5, 8, 10, 15, etc. seconds), which is represented by the time 309 in the time bar 308. Additionally or alternatively, in some examples, the electronic device 301 detects that another timing criterion is met (e.g., different from the above gaze-based timing criterion). For example, in Figure 3B , the electronic device 301 detects that the first area 316-1 of the document 315 remains in the user's field of view (and is therefore detectable by the one or more cameras of the electronic device 301) for at least a threshold amount of time (e.g., 1, 2, 3, 5, 10, 15, 20, 30, etc. seconds).

[0049] In some examples, such as Figure 3BAs shown, in response to detecting a user's gaze 321 pointing to the first area 316-1 (e.g., for a threshold amount of time), the electronic device 301 displays the user interface element 320-1 in the three-dimensional environment 350. In some examples, as described above, the user interface element 320-1 includes suggested text (e.g., "Casey Williams") for assisting the user in providing (e.g., writing) information in the first area 316-1. In some examples, the electronic device 301 generates the suggested text in the user interface element 320-1 based on detecting the first area 316-1 of the document 315. For example, the electronic device 301 utilizes image recognition (such as object recognition, character recognition, computer vision, or any other suitable detection technology) to identify one or more characteristics of the first area 316-1. In some examples, the electronic device 301 displays the suggested text in the three-dimensional environment 350 without the user interface element 320-1 (e.g., the suggested text Casey Williams is not necessarily contained within the virtual element). In some examples, the electronic device 301 detects the presence of a line (or box or other spacing or blank (e.g., white space)) of the word "name" included in the first area 316-1 (e.g., a text description or label intended for the content of the first area 316-1) and a text input field indicating that text or other content can be entered therein (e.g., and that the text or other content has not yet been entered therein). In some examples, based on identifying the first area 316-1 as a text input field and the word "name", the electronic device 301 generates the suggested text Casey Williams, which, as discussed above, is optionally the name of the user of the electronic device 301. In some examples, the user interface element 320-1 is displayed in any of the locked orientations defined herein. For example, the user interface element 320-1 (including the various user interface elements discussed below) can be displayed in the three-dimensional environment 350 in a world locked orientation, a body locked orientation, or a head locked orientation.

[0050] In some examples, the electronic device 301 generates suggested text (e.g., Casey Williams) using data stored (e.g., in a memory) and accessible by the electronic device 301. In some examples, the data is stored by a user of the electronic device 301 (e.g., the user provides input detected by the electronic device 301 for storing data corresponding to the name Casey Williams). In some examples, the electronic device 301 generates suggested text using data provided by an application running on the electronic device 301. For example, the electronic device 301 accesses a repository or library of user data from a contact application (e.g., a digital phone book), a web browsing application, a health or fitness application, a media player application, a photo application, a note application, etc., which enables the electronic device 301 to generate suggested text corresponding to the detected word name. In some examples, the data stored and accessible by the electronic device 301 corresponds to information captured opportunistically by the electronic device 301 (e.g., without user input that causes the information to be captured by the electronic device 301). For example, if the electronic device 301 visually detects (e.g., via one or more cameras or other sensors) information determined to be relevant to the user in the physical environment, such as the license plate number of the user's vehicle, the user's driver's license number, etc., the electronic device 301 may capture an image and / or perform image recognition (e.g., optical character recognition) and store the information for generating suggested text. In some examples, the electronic device 301 generates suggested text based on the type of object detected by the electronic device 301. For example, as discussed above, the document 315 corresponds to a parking permit form, which indicates (e.g., based on detecting the "Parking Permit Form" title) that the information to be provided by the user involves the vehicle and other driving-related information associated with the user. In some examples, based on determining that the electronic device 301 has access to data related to the first area 316-1 (e.g., stored in a memory or provided by an application), the electronic device 301 uses the data discussed above to generate suggested text. For example, if the electronic device 301 determines that information corresponding to a name (e.g., the name of the user or the names of other users associated with the user of the electronic device 301) cannot be located, accessed, or otherwise known by the electronic device 301, the electronic device 301 abandons displaying the user interface element 320-1 in the three-dimensional environment 350.

[0051] In some examples, based on determining that suggested content can be provided to the user at the electronic device 301, the electronic device 301 provides suggested text via the user interface element 320-1. For example, providing suggested content based on detecting one or more areas of an object (e.g., document 315) in the physical environment 318 depends on whether a particular operating mode is enabled by the user (e.g., provided in a settings application on the electronic device 301). In some examples, the operating mode in which suggested content is provided can be enabled for specific types of objects (e.g., documents) detected by the electronic device 301 but not other types of objects (e.g., locked devices, personal identification number (PIN) entry devices, etc.). Therefore, if the user has opted in (e.g., provided user permission) and / or enabled providing suggested content, the electronic device 301 is optionally able to access data associated with the user (e.g., stored in memory or provided by one or more applications running on the electronic device 301 as discussed above).

[0052] In some examples, based on determining that the electronic device 301 is currently in an unlocked state (e.g., the electronic device 301 is not in sleep mode), the electronic device 301 provides suggested text via the user interface element 320-1. In some examples, based on determining that the user of the electronic device 301 has been authenticated, the electronic device 301 provides suggested text via the user interface element 320-1. For example, the electronic device 301 is in an unlocked state, and the identity and / or authorization of the user using the electronic device 301 has been verified using biometric authentication (e.g., fingerprint authentication, facial recognition, voice recognition, etc.). In some examples, user authentication is performed once (e.g., when the electronic device 301 is first powered on and / or accessed by the user (e.g., when the electronic device 301 is first positioned on the user's head)). In some examples, user authentication is performed each time a suggestion is generated for the user (e.g., before the user interface element 320-1 is displayed in the three-dimensional environment 350).

[0053] In some examples, the electronic device 301 generates suggested content based on the determined and / or predicted difficulty or complexity level of the content intended to be filled or entered into the corresponding area / field in the document 315. For example, as discussed above, the first area 316-1 includes a text description name, which indicates that text corresponding to a specific name (such as the user's name) is intended to be written after the text description in the first area 316-1. In some examples, not as discussed above and as Figure 3BAs shown, the suggested text Casey Williams is displayed in the three-dimensional environment 350. The electronic device 301 may abandon displaying the suggested text based on determining that the difficulty level of the content (e.g., name) intended to be entered into the first area 316-1 is relatively low. For example, the electronic device 301 determines and / or predicts that the user is able to enter a specific name into the first area 316-1 without the assistance of the electronic device 301 (e.g., based on the user's memory and / or cognitive function). On the other hand, for more difficult or complex areas (e.g., based on the length of the expected content), such as the fifth area 316-5 that prompts the user to enter a license plate number (or other similar difficult types of content, such as a vehicle identification number or a user identification number), the electronic device 301 may display suggested text in the three-dimensional environment 350 to assist the user in filling in the more difficult area. In some examples, the electronic device 301 generates the suggested text based on the confidence level determined by the electronic device 301 for a specific area (e.g., the first area 316-1). For example, the electronic device 301 displays suggested text in the user interface element 320-1 based on determining that the confidence for the first area 320-1 exceeds a threshold confidence (e.g., greater than 75%, 80%, 85%, or 90% confidence) based on one or more of the above factors.

[0054] In some examples, such as Figure 3B As shown, electronic device 301 displays user interface element 320-1 at a position in three-dimensional environment 350, the position being based on the position of first region 316-1 in document 315. For example, Figure 3B As shown, the user interface element 320-1 is displayed in the three-dimensional environment 350 at a location outside the border or perimeter of the first area 316-1 in the document 315 (e.g., above the first area 320-1 relative to the user's viewpoint). Additionally or alternatively, in some examples, the electronic device 301 displays the user interface element 320-1 at a location in the three-dimensional environment 350 that is at least a threshold distance (e.g., 0.1, 0.5, 1, 1.5, 2, 3, 4, etc. cm) from the first area 316-1 (e.g., relative to the center of the first area 316-1 or a specific portion of the first area 316-1, such as a name tag or a row adjacent to the name tag). In either case, the location at which the user interface element 320-1 is displayed in the three-dimensional environment 350 is different from the location corresponding to the first area 316-1 in the three-dimensional environment 350 (e.g., the locations do not overlap), which enables a user to easily input content (e.g., write text) into the first area 316-1 while the user interface element 320-1 is displayed (e.g., the user interface element does not obscure a field that the user intends to fill in).

[0055] Figure 3CAn example is shown in which the first region 316-1 discussed above is divided into or composed of a first sub-region 316-1a and a second sub-region 316-1b (e.g., below the first sub-region 316-1a or next to the first sub-region 316-1a) in the document 315. Figure 3C As shown, the name tag associated with the first region 316-1 discussed above is alternatively divided into a "first name" tag and a "last name" tag. In some examples, in response to detecting a gaze 321 directed toward the first sub-region 316-1a, as shown in FIG. Figure 3C As shown, electronic device 301 simultaneously displays first user interface element 320-1a and second user interface element 320-1b in three-dimensional environment 350. For example, because first sub-region 316-1a and second sub-region 316-1b share the same context (e.g., first name tag and last name tag jointly correspond to name), electronic device 301 simultaneously displays first user interface element 320-1a and second user interface element 320-1b in three-dimensional environment 350. Additionally or alternatively, in some examples, electronic device 301 simultaneously displays first user interface element 320-1a and second user interface element 320-1b in three-dimensional environment 350 because the user intent to interact with document 315 is detected as being directed to both first sub-region 316-1a and second sub-region 316-1b. For example, electronic device 301 detects gaze 321 directed to first sub-region 320-1a, while an object associated with the user (e.g., finger, pen, pencil, etc.) is directed to second sub-region 320-1b (as discussed in more detail later herein).

[0056] In some examples, such as Figure 3C As shown, first user interface element 320-1a and second user interface element 320-1b include suggestion text corresponding to the suggestion text provided in user interface element 320-1 discussed above. For example, first user interface element 320-1a provides suggestion text indicating the user's first name (e.g., "Casey"), and second user interface element 320-1b provides suggestion text indicating the user's last name (e.g., "Williams"). Figure 3C As shown and as similarly discussed above, the first user interface element 320-1a is displayed at a first position based on a position corresponding to the first sub-area 316-1a in the three-dimensional environment 350, and the second user interface element 320-1b is displayed at a second position based on a position corresponding to the second sub-area 316-1b in the three-dimensional environment 350.

[0057] In some examples, a user may express an intention to interact with a specific area in document 315 via an object (e.g., a writing tool) detectable by electronic device 301. Figure 3DAs shown, the user is holding (e.g., in hand 306) a writing instrument 303 (e.g., a pen, pencil, marker, etc.) that is visible in the user's field of view in physical environment 318. In some examples, electronic device 301 visually detects (e.g., via one or more cameras of electronic device 301) that writing instrument 303 is in contact with document 315 (or the tip of writing instrument 303 is within a threshold distance from the surface of document 315, such as 0.05, 0.1, 0.25, 0.5, 1, 2, etc. cm). Specifically, in Figure 3D , the electronic device 301 detects that the writing instrument 303 is in contact with the document 315 at a portion of the third area 316-3 discussed previously above. In some examples, the electronic device 301 detects that the writing instrument 303 is in contact with the document 315 for at least a threshold amount of time, such as the threshold amount of time discussed previously above.

[0058] In some examples, such as Figure 3D As shown, in response to detecting contact of the writing tool 303 on the document 315 directed toward the third area 316-3, the electronic device 301 generates suggested text corresponding to the third area 316-3. For example, as similarly discussed above, the electronic device 301 identifies one or more characteristics of the third area 316-3, such as the label “mobile phone” in the third area 316-3. In some examples, based on the label “mobile phone” identified in the third area 316-3, the electronic device 301 determines one or more mobile phone numbers associated with the user of the electronic device 301. In some examples, based on determining that more than one mobile phone number is identified as being associated with the user of the electronic device 301 (e.g., based on data stored in a memory or provided by one or more applications running on the electronic device 301, as similarly discussed above), the electronic device 301 presents multiple user interface elements in the three-dimensional environment 350. For example, as Figure 3D As shown, electronic device 301 simultaneously displays third user interface element 320-2a and fourth user interface element 320-2b in three-dimensional environment 350. In some examples, such as Figure 3D As shown, the third user interface element 320-2a includes suggested text corresponding to a first mobile phone number associated with the user, and the fourth user interface element 320-2b includes suggested text corresponding to a second mobile phone number associated with the user. Figure 3D The suggestion text in is alternatively presented as a scrollable list (e.g., a horizontally scrollable row rather than a vertical list).

[0059] In some examples, the user may express an intention to interact with a particular area in the document 315 via a gesture that is performed using a portion of the user (e.g., a hand) that can be detected by the electronic device 301. Figure 3E As shown, the user's hand 306 is currently visible in the portion of the physical environment 318 included in the three-dimensional environment 350. In some examples, such as Figure 3E As shown, electronic device 301 detects a gesture of pointing to document 315 performed by hand 306 in physical environment 318. Figure 3E In some examples, the electronic device 301 detects that a finger (e.g., index finger) of the hand 306 is in contact with the document 315 (or the tip of the finger is within a threshold distance, such as 0.05, 0.1, 0.25, 0.5, 1, 2, etc. cm, from the surface of the document 315). Figure 3E As shown, the electronic device 301 detects that the finger is in contact with the document 315 at a position corresponding to the fourth area 316-4 discussed previously above. Alternatively, in some examples, the electronic device 301 detects a "pointing gesture" in which the index finger of the hand 306 extends outward and points to the fourth area 316-4 of the document 315 (regardless of whether it is necessary to detect that the index finger of the hand 306 is in contact with the surface of the document 315). In some examples, the electronic device 301 detects an air pinch gesture (e.g., in which the index finger and thumb of the hand 306 are brought together to make contact) while the user's gaze is directed to the fourth area 316-4.

[0060] In some examples, such as Figure 3E As shown, in response to detecting a gesture directed to the fourth area 316-4 by the user's hand 306, the electronic device 301 generates suggested text to assist the user in entering text into the fourth area 316-4, as similarly discussed above. For example, as discussed above, the electronic device 301 recognizes the text label "Brand / Model" in the fourth area 316-4, and based on data stored at the electronic device 301 or provided by one or more applications running on the electronic device 301, generates suggested text "Model A Brand A", as similarly discussed above herein. In some examples, as Figure 3E As shown, the suggested text is displayed in the user interface element 320-3 in the three-dimensional environment 350. In some examples, as similarly discussed above, the electronic device 301 displays the user interface element 320-3 at a position in the three-dimensional environment 350 that is based on a position corresponding to the fourth area 316-4 in the document 315 (e.g., outside the position corresponding to the fourth area 316-4), such as Figure 3E shown.

[0061] exist Figure 3F In FIG. 3 , electronic device 301 detects handwritten text 322-1 in first area 316-1 of document 315 in physical environment 318. For example, Figure 3FAs shown, electronic device 301 visually detects (e.g., via one or more cameras) the text "Jacob Smith" written by a user in the user's field of view using writing instrument 303 (e.g., held in the user's hand 306). In some examples, Figure 3F , the electronic device 301 detects that the tip of the writing tool 303 is still in contact with the document 315 (e.g., the user is still writing the handwritten text 322-1). In some examples, when the electronic device 301 detects the handwritten text 322-1 (e.g., Jacob Smith), the electronic device 322-1 stores the data corresponding to the text (e.g., the name Jacob Smith) for later generation and display of suggested text to the user. For example, as similarly discussed above, if the electronic device 301 recognizes the text tag "name" in a subsequent document in response to detecting one of the triggers discussed above, the electronic device 301 may generate and display "Jacob Smith" as a suggestion to the user.

[0062] exist Figure 3G In FIG. 3 , electronic device 301 detects that the tip of writing tool 303 is lifted off the surface of document 315. For example, Figure 3G As shown, the tip of writing instrument 303 is no longer in contact with document 315 , and the user has finished writing handwritten text 322 - 1 into first area 316 - 1 of document 315 .

[0063] In some examples, in response to detecting that the tip of the writing instrument 3-3 is lifted off the surface of the document 315, Figure 3G As shown, the electronic device 301 generates suggested text for a subsequent (eg, subsequent) region in the document 315. For example, Figure 3G As shown, the second region 316-2 is spatially after (e.g., located below) the first region 316-1, and is therefore determined by the electronic device 301 to be a subsequent region in the document 315. In some examples, the electronic device 301 determines the subsequent region based on the reading language of the text in the document 315. For example, based on the title of the document 315 (parking permit form) and / or the text of the plurality of regions 316, the electronic device 301 determines that the document 315 is read from left to right (e.g., because English is a language that is read from left to right and from top to bottom). Therefore, after reading the English text in the document 315 from left to right, the electronic device 301 determines that the second region 316-2 is a subsequent region in the document 315.

[0064] In some examples, such as Figure 3GAs shown, the electronic device 301 generates suggested text for the second region 316-2 in the document 315 (e.g., a region subsequent to the first region 316-1 as discussed above). For example, as similarly discussed herein, the electronic device 301 generates the suggested text "019000126" based on detecting the text label "driver's license number" in the second region 316-2 in the document 315. In some examples, as Figure 3G As shown, the suggested text is displayed in the user interface element 320-4 in the three-dimensional environment 350. In some examples, the electronic device 301 displays the user interface element 320-4 in the three-dimensional environment 350 regardless of whether the user's gaze 321 is directed to the second area 316-2 of the document 315 in the portion of the physical environment 318 included in the three-dimensional environment 350. For example, Figure 3G As shown, although the gaze 321 is directed to the second area 316-2 in the document 315, the electronic device 301 will display the user interface element 320-4 of the second area 316-2 even if the gaze 321 is detected as being directed to an area different from the second area 316-2. In some examples, as similarly discussed above, the electronic device 301 displays the user interface element 320-4 at a position in the three-dimensional environment 350 that is based on the position corresponding to the second area 316-2 (but does not overlap with the position corresponding to the second area 316-2).

[0065] exist Figure 3H In FIG. 3 , the electronic device 301 detects handwritten text 322-2 in the second area 316-2 of the document 315. For example, Figure 3H As shown, electronic device 301 detects that the text "019000126" (e.g., corresponding to the suggested text in user interface element 320-4) has been written in second area 316-2 by the user using writing tool 303 (e.g., held in hand 306). Figure 3H In the example of , electronic device 301 detects that the tip of writing instrument 303 is still in contact with the surface of document 315 .

[0066] exist Fig. 3I In FIG. 3 , electronic device 301 detects that writing tool 303 is lifted off the surface of document 315 in the user's field of view. Fig. 3I As shown, electronic device 301 detects that the tip of writing tool 303 no longer contacts the surface at second area 316-2 of document 315. In some examples, such as Fig. 3IAs shown, in response to detecting that the writing tool 303 is lifted off from the document 315, the electronic device 301 stops displaying the user interface element 320-4 in the three-dimensional environment 350. In some examples, the electronic device 301 stops displaying the user interface element 320-4 in the three-dimensional environment 350 in response to detecting the handwritten text 322-2 in the second area 316-2 of the document 315. For example, the electronic device 301 determines that the handwritten text 322-2 in the second area 316-2 corresponds to the suggested text in the user interface element 320-4 (regardless of whether it is necessary to detect that the writing tool 303 is lifted off from the document 315), thereby indicating that the user interface element 320-4 is no longer needed. Alternatively, in some examples, the electronic device 301 stops displaying the user interface element 320-4 after determining that a threshold amount of time (e.g., 1, 5, 10, 15, 20, 30, 60, etc. minutes) has passed since the user interface element 320-4 was displayed. As another example, electronic device 301 stops displaying user interface element 320-4 after detecting a user intent to interact with an area different from second area 316-2, such as detecting gaze 321 or a writing instrument's touch screen directed toward third area 316-3 in physical environment 318.

[0067] In some examples, such as Fig. 3I As shown, after detecting that the writing tool 303 is lifted off the document 315 and / or directed to the gaze 321 of the third area 316-3 of the document 315, the electronic device 301 generates suggested text for assisting the user to enter text into the third area 316-3. For example, as similarly discussed above, the electronic device 301 detects the text label "mobile phone" in the third area 316-3. In some examples, the electronic device 301 adjusts and / or updates the suggested text provided to the user based on text (e.g., handwritten text) entered by the user in other (e.g., previous) areas of the document 315. For example, as discussed above, in Fig. 3I , the user has written handwritten text 322-1 into the first area 316-1 of the document 315 and has written handwritten text 322-2 into the second area 316-2 of the document 315. Therefore, the electronic device 301 generates suggested text for the detected third area 316-3 based at least in part on the handwritten text 322-1 and / or the handwritten text 322-2. In some examples, the electronic device 301 determines that the information provided on the document 315 does not correspond to the user of the electronic device 301. For example, as previously discussed above, the user of the electronic device 301 is Casey Williams. Fig. 3IAs shown, the information entered in document 315 is for Jacob Smith, a different person than Casey Williams (e.g., because Jacob Smith is currently using electronic device 301 or Casey Williams is filling out a parking permit form for Jacob Smith). Therefore, electronic device 301 optionally generates suggested text corresponding to Jacob Smith's known mobile phone number instead of Casey Williams' known mobile phone number (e.g., Casey Williams' known mobile phone number was previously entered in the document 315). Figure 3D ).

[0068] In some examples, such as Fig. 3I As shown, the electronic device 301 displays a user interface element 320-5 including a suggested text generated based on at least the handwritten text 322-1 and / or 322-2 in the three-dimensional environment 350. Fig. 3I As shown, the electronic device 301 generates the suggested text "(101)788-1111" based on the handwritten text Jacob Smith (e.g., the suggested text corresponds to a known mobile phone number of Jacob Smith). In some examples, as previously described above, based on determining that the electronic device 301 cannot locate and / or identify the suggested text based on the name Jacob Smith, the electronic device 301 abandons displaying the user interface element 320-5 in the three-dimensional environment 350 (or displays a known mobile phone number, such as a mobile phone number associated with a user of the electronic device 301). Alternatively, in some examples, the electronic device 301 may display a visual indication that the suggested text (or other content) cannot be generated and / or that additional information is required to generate the suggested text. In some examples, as similarly discussed herein, the electronic device 301 optionally displays the user interface element 320-5 at a location in the three-dimensional environment 350 that is based on a location corresponding to the third area 316-3 of the document (e.g., above the third area 316-3).

[0069] exist Figure 3J In FIG. 3 , the user of electronic device 301 has provided additional handwritten text in document 315 in physical environment 318. For example, Figure 3J As shown, the user has provided handwritten text 322-3 in the third area 316-3 of the document 315 (e.g., corresponding to the suggested text in the user interface element 320-5 discussed above) and provided handwritten text 322-4 in the fourth area 316-4 of the document 315 (e.g., corresponding to Figure 3E In addition, in Figure 3J, the electronic device 301 detects contact of the writing tool 303 (eg, held in the user's hand 306) on the document 315. Specifically, as similarly discussed above, the electronic device 301 detects that the tip of the writing tool 303 is at the fifth area 316-5 on the document 315.

[0070] In some examples, such as Figure 3J As shown, in response to detecting that the writing tool 303 is at the fifth area 316-5 on the document 315, the electronic device 301 generates suggested text for assisting the user to enter text into the fifth area 316-5, as similarly discussed above. Figure 3J As shown, electronic device 301 displays user interface element 320-6, which includes the suggested text "7TYP290" corresponding to a known license plate number (e.g., the license plate number of the user's car and / or the car of another user associated with the user (such as John Smith as discussed above). Figure 3J As shown, the user interface element 320 - 6 is displayed in the three-dimensional environment 350 at a location that is based on a location corresponding to the fifth region 316 - 5 of the document 315 (eg, outside and / or above the fifth region 316 - 5 ).

[0071] exist Figure 3K In FIG. 3 , the electronic device 301 detects handwritten text 322-5 in the fifth area 316-5 of the document 315. For example, Figure 3K As shown, the user has used the writing tool 303 to write the text "7TYP390" in the fifth area 316-5 of the document 315, which is currently visible in the user's field of view.

[0072] In some examples, the electronic device 301 may be configured to detect errors (e.g., word order errors, grammatical errors or inconsistencies, word usage errors, etc.) present in the handwritten text provided in the document 315. In some examples, the electronic device 301 detects errors based on a comparison between the handwritten text and the suggested text generated by the electronic device 301. For example, Figure 3KAs shown, the electronic device 301 detects an error in the handwritten text 322-5 in the fifth area 316-5 of the document 315. Specifically, the electronic device 301 recognizes (e.g., via character recognition) that the number "2" in the suggested text in the user interface element 320-6 has been mistakenly written by the user as the number "3" in the handwritten text 322-5 (e.g., based on the comparison between 7TYP390 and 7TYP290). In some examples, the electronic device 301 may be configured to detect errors in the handwritten text regardless of whether the suggested text (e.g., such as the suggested text in the user interface element 320-6 discussed above) has been previously displayed in the three-dimensional environment 350. For example, the electronic device 301 is configured to use a spelling checker and / or a grammar checker running on the electronic device 301 to detect word order errors or other grammatical errors. Alternatively, in some examples, the electronic device 301 is configured to detect errors in the handwritten text only when the suggested text has been previously displayed (and / or is currently displayed) in the three-dimensional environment 350.

[0073] In some examples, such as Figure 3K As shown, when the electronic device 301 determines that the handwritten text 322-5 includes an error as discussed above, the electronic device 301 presents a warning indicating that the handwritten text 322-5 includes an error. Figure 3K As shown, electronic device 301 displays message 323 in three-dimensional environment 350, which alerts the user of electronic device 301 of the detected error (e.g., "incorrect input"). Additionally or alternatively, in some examples, electronic device 301 provides a visual indication of the error detected in handwritten text 322-5 in three-dimensional environment 350. For example, Figure 3K As shown, the electronic device 301 displays an indication 327 (e.g., a jagged or eroded underline) below the portion of the handwritten text 322-5 containing an error, specifically below the number "3" in the handwritten text 322-5. In some examples, such as Figure 3K As shown, when the electronic device 301 detects an error in the handwritten text 322-5, the electronic device 301 updates the display of the user interface element 320-6 to include an indication 326 for assisting the user in correcting the error detected in the handwritten text 322-5. Specifically, as shown in FIG. Figure 3K As shown, electronic device 301 optionally displays indication 326 (e.g., underlining) below the portion of the suggestion text corresponding to the error, such as below the number "2" in the suggestion text in user interface element 320-6. Additionally or alternatively, in some examples, such as Figure 3K As shown, the electronic device 301 displays the portion of the suggested text corresponding to the error with visual emphasis relative to other portions of the suggested text to instruct the auxiliary user to correct the error in the handwritten text 322-5. Figure 3K , the electronic device 301 bolds or highlights the number “2” in the suggested text in the user interface element 320 - 6 or changes its color.

[0074] In some examples, such as Figure 3K As shown, the electronic device 301 displays the portion of the suggested text that is detected as having been correctly written by the user (e.g., a portion of the handwritten text 322-5 matches a portion of the suggested text) with one or more indications, wherein the one or more indications visually indicate that the corresponding text in the handwritten text 322-5 is detected as correct. Figure 3K , the electronic device 301 displays an indication 328 (e.g., a dotted or patterned underline) in the user interface element 320-6 below the portion of the suggested text that is detected as being correctly written. Alternatively, in some examples, the electronic device 301 updates the visual appearance or visual emphasis of the portion of the suggested text that is detected as being correctly written within the handwritten text 322-5. For example, the electronic device 301 may display the characters "7TYP" and "90" in a particular color (e.g., green, blue, etc.), shaded, highlighted, or in other patterned ways to indicate that the user has correctly written those portions of the suggested text in the handwritten text 322-5. In some examples, as the user writes the handwritten text 322-5, the electronic device 301 updates the display of each character of the suggested text based on whether the suggested text is detected as correct or incorrect based on each character of the handwritten text 322-5. For example, in Figure 3KIn the example, when the user writes the character "7" in the handwritten text 322-5, the electronic device 301 updates the corresponding character "7" in the suggested text of the user interface element 320-6 to have a first color (e.g., blue), because the handwritten character "7" is detected as correct, and when the user later writes the character "3" in the handwritten text 322-5, the electronic device 301 updates the corresponding character "2" in the suggested text of the user interface element 320-6 to have a second color (e.g., red), because the handwritten character "2" is detected as incorrect, as discussed similarly above. In some such examples, if the user rewrites one or more portions of the handwritten text 322-5 to correct the detected error (e.g., erase and / or remove the character "3" and write the character "2"), the electronic device 301 updates the display of the suggested text in the user interface element 320-6 to have a third color (e.g., green), which indicates that the entire suggested text has been correctly written by the user. In some examples, the verification of the input is performed on a per-character basis. For example, the system causes the suggested text to change from a first color to a second color for each character entered correctly and / or causes the suggested text to change from a first color to a third color for each character entered incorrectly. In some examples, verification of the input is additionally or alternatively performed on a per-word basis or after completing the input of the suggested text (e.g., a complete license plate). For example, in addition to indicating correct or incorrect input on a per-character basis, once all characters in a word are correctly entered, the electronic device 301 may cause the suggested text to change the characters that make up the word from a first color, a second color, or a third color to a fourth color. In some examples, the indication of verification of the input is part of an animation that stops displaying the suggested text.

[0075] Additionally or alternatively, in some examples, the electronic device 301 displays an indication 329 of the portion of the handwritten text 322-5 that is detected as having been correctly written by the user. Figure 3K As shown, the electronic device 301 displays a double underline below the portion of the handwritten text 322 - 5 that corresponds to and matches the suggested text in the user interface element 320 - 6 .

[0076] As discussed above, the electronic device 301 optionally determines that the handwritten text 322-5 includes an error based on a comparison between the handwritten text 322-5 and the suggested text in the user interface element 320-6. Specifically, the electronic device 301 determines that the handwritten text 322-5 corresponds to the suggested text in the user interface element 320-6 (e.g., based on detecting the same characters (e.g., numbers and letters) above a threshold and in the same order (e.g., more than three, four, five, or six characters are the same and appear in the same order between the handwritten text 322-5 and the suggested text)). Therefore, the electronic device 301 recognizes based on the comparison that the number "3" has been written in the position where the number "2" should be in the handwritten text 322-5, as discussed above. In some examples, if the electronic device 301 determines that the difference between the handwritten text 322-5 and the suggested text in the user interface element 320-6 exceeds a threshold amount (e.g., because the handwritten text 322-5 does not correspond to the suggested text), the electronic device 301 abandons displaying the visual indications discussed above (e.g., message 323, indication 327, indication 326, indication 328, and / or indication 329) in the three-dimensional environment 350.

[0077] In some examples, after detecting that writing instrument 303 is lifted off document 315, electronic device 301 displays a visual indication that alerts the user to the detected error in handwritten text 322-5. Figure 3K In some examples, electronic device 301 displays message 323, indication 327, indication 328, and / or indication 326 in three-dimensional environment 350 in response to detecting that the user is no longer writing text in document 315 (e.g., using writing tool 303). Alternatively, in some examples, electronic device 301 displays a visual indication in real time alerting the user of the error detected in handwritten text 322-5 (e.g., regardless of whether the user is still writing text in document 315). It should be understood that although indications 326, 327, 328, and 329 are in Figure 3K , and described above as being presented as an underline, but alternative forms of indication may be provided, such as highlighting, changing color, circling, boxing, strikethrough, etc.

[0078] As previously referenced above Figure 3E As described, the electronic device 301 optionally generates suggested text in response to detecting a gesture performed by a user's hand and pointing to a specific area of ​​the document 315 in the three-dimensional environment 350. Specifically, as discussed above, the electronic device 301 displays a user interface element including suggested text in the three-dimensional environment 350 in response to detecting that a finger of the hand (e.g., index finger) is in contact with the document 315 (or the tip of the finger is within a threshold distance from the surface of the document 315, such as 0.05, 0.1, 0.25, 0.5, 1, 2, etc. cm).

[0079] In some examples, such as Figure 3L As shown, the electronic device 301 detects a gesture and / or posture (e.g., a pointing gesture or posture using an index finger) performed by the user's hand 306 pointing to the fourth area 316-4 of the document 315 in the three-dimensional environment 350, as similarly discussed above. Figure 3L As shown, the electronic device 301 detects that the tip of the index finger of the hand 306 points to the text label "Brand / Model" in the fourth area 316-4 (e.g., the tip of the index finger is positioned below the word "Brand / Model" and / or above / on the word "Brand / Model" without having to contact the surface of the document 315, as similarly discussed above). In some examples, in addition to detecting the index finger, the electronic device 301 also detects a pointing gesture of the hand 306 (e.g., the electronic device 301 detects that the other fingers of the hand 306 are curled inward toward the palm of the hand 306).

[0080] In some examples, in response to detecting a gesture directed to the fourth area 316-4 performed by the user's hand 306, the electronic device 301 generates suggested text to assist the user in entering text into the fourth area 316-4 based on determining that the gesture is detected (e.g., the hand 306 maintains the gesture) for at least a threshold amount of time (such as 0.5, 1, 1.5, 2, 3, 5, 8, 10, etc. seconds). In some examples, in response to detecting a gesture directed to the fourth area 316-4 performed by the hand 306, the electronic device 301 displays a visual indicator 335 associated with the threshold amount of time, which is determined by Figure 3L The time 341 in the time bar 340 in FIG. Figure 3L As shown, electronic device 301 displays a representation of a hand, which is displayed together with an animation indicating the passage of a threshold amount of time, such as visual filling and / or filling visual indicator 335, as represented by shading 336. Figure 3L In some examples, visual indicator 335 is filled / filled (e.g., via shading 336) in three-dimensional environment 350 by an amount that corresponds to (e.g., is equal to or proportional to) an amount of time that has elapsed since the gesture performed by hand 306 was detected, as represented in time bar 340. In some examples, visual indicator 335 is displayed at a location in three-dimensional environment 350 based on hand 306 (e.g., the location of hand 306 in three-dimensional environment 350). For example, as shown in FIG. Figure 3LAs shown, electronic device 301 displays visual indicator 335 at a position adjacent to hand 306 (e.g., the portion of hand 306 visible via transmittal) in three-dimensional environment 350 (e.g., the right side of the hand). Alternatively, in some examples, visual indicator 335 is displayed at a predetermined position in three-dimensional environment 350, such as a position corresponding to the center of the field of view of electronic device 301 or a position corresponding to a corner or edge of the display of electronic device 301. Figure 3L In the example of , because a threshold amount of time has not elapsed since the gesture performed by hand 306 was detected, electronic device 301 abandons displaying a user interface element including suggested text in three-dimensional environment 350.

[0081] In some examples, such as Figure 3M As shown, based on determining that a threshold amount of time represented by time 341 in time bar 340 has elapsed since a gesture performed by hand 306 was detected and hand 306 has maintained the gesture for a threshold amount of time, electronic device 301 displays a user interface element 320-3 including suggested text (e.g., "Model A Brand A") for assisting a user in entering text into fourth area 316-4. In some examples, user interface element 320-3 corresponds to the above reference 306. Figure 3E The user interface element 320-3 in question. In some examples, Figure 3M As shown, because the threshold amount of time described above has elapsed, the visual indicator 335 (e.g., a representation of a hand) is updated in the three-dimensional environment 350 to visually appear to be completely filled / filled, thereby visually indicating that the user has maintained the gesture for the amount of time required to display the user interface element 320-3 in the three-dimensional environment 350. In some examples, the electronic device 301 stops displaying the visual indicator 335 after detecting that the hand 306 is no longer performing the gesture discussed above (e.g., the index finger of the hand 306 is no longer pointing to and / or contacting the text label brand / model in the fourth area 316-4). In some examples, the electronic device 301 stops displaying the visual indicator 335 after displaying the user interface element 320-3 in the three-dimensional environment 350. For another example, the electronic device 301 may stop displaying the visual indicator 335 in the three-dimensional environment 350 based on determining that the user has correctly entered / written text in the fourth area 316-4 relative to the suggested text of the user interface element 320-3, as described above with reference to FIG. Figure 3K Similar discussed.

[0082] In some examples, if electronic device 301 detects that hand 306 stops performing a gesture using hand 306 pointing to document 315 before a threshold amount of time represented by time 341 has elapsed, electronic device 301 abandons displaying user interface element 320-3 in three-dimensional environment 350. Additionally, in such cases, electronic device 301 optionally stops displaying visual indicator 335 in three-dimensional environment 350, thereby visually indicating that the user has not maintained the gesture using hand 306 for the amount of time required to display user interface element 320-3 in three-dimensional environment 350. Additionally, in such examples, if the user were to re-perform the gesture discussed above using hand 306 pointing to document 315, electronic device 301 may restart tracking the elapse of the threshold amount of time (e.g., starting from zero), rather than starting from the instance relative to the threshold amount of time at which the user stopped performing the gesture. As an alternative example, if electronic device 301 detects that hand 306 stops performing the gesture and / or posture of pointing to document 315 using hand 306 before the threshold amount of time discussed above has elapsed, electronic device 301 reverses the animation of visual indicator 335 (e.g., represented by shading 336). For example, electronic device 301 reverses the filling / filling of visual indicator 335 so that visual indicator 335 appears to be empty in three-dimensional environment 350, during which time the user may re-perform the gesture discussed above to resume the elapse of the threshold amount of time required to display user interface element 320-3 in three-dimensional environment 350. In some such examples, the timing of the emptying of visual indicator 335 may be different from the timing associated with the filling / filling of visual indicator 335 (e.g., which is equal to the threshold amount of time as discussed above). For example, electronic device 301 may clear visual indicator 335 faster than visual indicator 335 fills in three-dimensional environment 350 (or vice versa) via animation of visual indicator 335 (eg, represented by shading 336), or utilize some other timing mechanism.

[0083] In some examples, the electronic device 301 displays the user interface element 320-3 at a predetermined location in the three-dimensional environment 350, rather than at a location based on the fourth area 316-4 as previously discussed above. Figure 3M As shown, the electronic device 301 displays the user interface element 320-3 at a location corresponding to the center of the field of view of the electronic device 301, rather than at a location above and / or adjacent to the fourth area 316-4 in the three-dimensional environment 350. In some such examples, the user interface element 320-3 is displayed as a world-locked object in the three-dimensional environment 350.

[0084] In some examples, the suggested text of user interface element 320-3 is displayed together with a visual indication of the information source used to generate the suggested text. For example, as previously discussed above, the suggested text Model A Brand A in user interface element 320-3 can be generated based on application data, user data stored in memory (e.g., in a file) and / or otherwise accessible to electronic device 301 (e.g., in a cloud database or on a second electronic device). In some examples, such as Figure 3M As shown, user interface element 320-3 may include the name, title, label, etc. of the information source, such as the name and / or icon corresponding to a specific application that provides data used to generate the suggestion text model A brand A and / or a file name stored in the memory for the electronic device 301 to use to generate the suggestion text (e.g., "found in file"). In some examples, such as Figure 3M As shown, a visual indication of the information source can be displayed below the suggested text in the user interface element 320-3. In addition, in some examples, the electronic device 301 can display the suggested text and additional information based on the context of the fourth area 316-4 (e.g., the text label brand / model). For example, Figure 3M As shown, user interface element 320-3 includes information corresponding to a vehicle identification number (VIN) associated with model A brand A. In some examples, the suggested text model A brand A can be visually depicted from other information (e.g., VIN and information source) included in user interface element 320-3 to visually and / or clearly indicate to the user the suggested text that has been generated by electronic device 301 for filling fourth area 316-4. For example, Figure 3M As shown, the suggested text Model A Brand A may be displayed in a maximum size and / or in bold within the user interface element 316 - 4 , while the VIN and source of information may be displayed in a smaller size and / or in non-bold within the user interface element 316 - 4 .

[0085] It should be understood that visual indicator 335 as described above is exemplary, and that visual indicator 335 may have an alternative appearance when displayed in three-dimensional environment 350. For example, visual indicator 335 may alternatively be displayed as a circular object, such as a dot, circle, sphere, or the like, or as any other suitable shaped object in three-dimensional environment 350. It should be understood that in such cases, the presentation of the animation that fills / populates visual indicator 335 (e.g., similar to shadow 336) may be maintained for the alternative form of visual indicator 335.

[0086]

[0013] Attention is now directed to additional examples of visually detecting one or more regions of a physical object in a physical environment and displaying one or more user interface elements including suggested content in a three-dimensional environment.

[0087] FIG. 4A to FIG. 4C An example of an electronic device displaying a user interface element including suggested text based on detecting one or more areas of a physical object according to some examples of the present disclosure is shown. In some examples, electronic device 401 has one or more characteristics of electronic device 301 discussed above. For example, electronic device 401 can be a head-mounted system / device and / or a projection-based system / device (including a hologram-based system / device) configured to generate and present a three-dimensional environment. FIG. 4A to FIG. 4C In an example of , the user optionally wears the electronic device 401 so that the three-dimensional environment 450 (e.g., a computer-generated environment) can be defined by the X, Y, and Z axes viewed from the perspective of the electronic device (e.g., a viewpoint associated with the user of the electronic device 401), as similarly discussed above. In some examples, the three-dimensional environment 450 has one or more characteristics of the three-dimensional environment 350 discussed above.

[0088] like Figure 4A As shown, electronic device 401 may be positioned in a physical environment 418 including a plurality of real-world objects. Figure 4A In the example, the physical environment 418 includes a second electronic device 400. The second electronic device 400 may be similar to the electronic device 201 discussed above, and / or may be a mobile device such as a smart phone, a tablet computer, or a laptop or desktop computer. In some examples, the second electronic device 400 includes a second electronic device similar to the electronic device 201 discussed above. Figure 2 One or more components of the electronic device 201 in the embodiment of the present invention, such as one or more touch-sensitive surfaces, one or more microphones or other audio sensors, one or more display generation components, one or more speakers, one or more processors, one or more memories and / or communication circuits. Figure 4A As shown, the second electronic device 400 includes a touch screen 420 configured to display one or more user interfaces and detect touch-based inputs. In some examples, the physical environment 418 has one or more characteristics of the physical environment 318 as described above.

[0089] In some examples, as similarly discussed above, the three-dimensional environment 450 presented using the electronic device 401 optionally includes a captured portion of the physical environment 418 surrounding the electronic device 401, such as a representation of the second electronic device 400 in the field of view of the three-dimensional environment 450. Figure 4A As shown, the second electronic device 400 is displaying a user interface 430 on the touch screen 420, as discussed in more detail below. Figure 4AAs shown, three-dimensional environment 450 may include representations of user interface 430 displayed on second electronic device 400 in physical environment 418. In some examples, these representations may include portions of the physical environment viewed through a transparent or semi-transparent display of electronic device 401.

[0090] In some examples, as described above, the second electronic device 400 is displaying a user interface 430. In some examples, the user interface 430 corresponds to a digital document (e.g., a digital form) that includes multiple areas (e.g., fillable fields) into which content such as text, images, drawings, etc. can be entered. Figure 4A As shown, user interface 430 includes multiple areas 416 configured to receive content (e.g., the multiple areas 416 can be selected to enter text or other content into each area in user interface 430). In some examples, the digital document of user interface 430 corresponds to a medical history form that prompts the user of electronic device 401 to provide information related to the medical history of a particular patient, such as the user or another person associated with the user of electronic device 401 (e.g., a spouse or partner, a child or other family member, a friend, etc.). In some examples, the user interface 430 includes a first area (416-1) in which the patient's name is indicated, a second area (416-2) in which the patient's date of birth is indicated, a third area (416-3) in which medications currently prescribed and / or consumed by the patient (if any) are indicated, a fourth area (416-4) in which allergens the patient has (if any) are indicated, a fifth area (416-5) in which surgeries or other major procedures the patient has undergone (if any) are indicated, a sixth area (416-6) in which the patient's family medical history, such as a history of disease, cancer, mental health disorders, or other conditions, is indicated, a seventh area (416-7) in which the patient's reason for visiting the medical facility is indicated, and an eighth area (416-8) in which the patient's insurance information is indicated. Figure 4A , the eighth area 416 - 8 is optionally outside the user's field of view of the three-dimensional environment 450 and is therefore optionally not visually detectable by the electronic device 401 .

[0091] exist Figure 4A , the user has entered text into one or more areas of user interface 430. For example, Figure 4A As shown, the first region 416-1 includes text 425-1 indicating the name of the patient associated with the medical history information in the user interface 430 (e.g., "Casey Williams"). In some examples, Casey Williams corresponds to the name of the user of the electronic device 401. Figure 4AAs shown, the second area 416-2 includes text 425-2 indicating the patient's date of birth associated with the medical history information in the user interface 430 (e.g., "01 / 12 / 1990"). In some examples, the text 425-1 and 425-2 are entered into the first area 416-1 and the second area 416-2, respectively, via a numeric keypad of the second electronic device 400 (e.g., in response to the second electronic device 400 detecting a tap of a contact (e.g., a finger, stylus, or other object) on the touch screen 420 directed to the first area 416-1 and the second area 416-2). Alternatively, in some examples, the text 425-1 and 425-2 are entered into the first area 416-1 and the second area 416-2, respectively, in response to the second electronic device 400 detecting (e.g., via one or more microphones) an audio-based input (e.g., dictation).

[0092] In some examples, as described above with reference to FIG. 3A to FIG. 3M Similarly discussed, it may be advantageous to provide a method for suggesting content based on detecting one or more areas of the user interface 430 displayed at the second electronic device 400 in the physical environment 418 to assist the user in inputting content into the one or more areas, rather than relying solely on such methods at the second electronic device 400. For example, the second electronic device 400 may be an electronic device that is not configured to (e.g., automatically) suggest content to the user based on the areas in the user interface 430. As another example, the second electronic device 400 may be an electronic device that does not belong to the user of the electronic device 401 and therefore has no access to the user data of the user of the electronic device 401 (e.g., in a memory or application running on the second electronic device 400). For example, as discussed above, the user interface 430 corresponds to a medical history form, and the user may fill out the medical history form in the doctor's office via the second electronic device 400 belonging to the doctor's office. Additionally or alternatively, the user fills out the medical history form of the user interface 430 for another person whose information is not stored on the second electronic device 400 and / or cannot be accessed by the second electronic device (while the user's information is stored and / or accessible).

[0093] exist Figure 4A In FIG. 4 , the electronic device 401 detects the user's gaze 421a in the three-dimensional environment 450, which points to the fourth area 416-4 of the user interface 430. For example, Figure 4A As shown, the electronic device 401 detects a gaze 421a directed to a location in the three-dimensional environment 450, which corresponds to the location of the fourth area 416-4 displayed by the second electronic device 400. Alternatively, in some examples, the electronic device 401 detects a gesture performed by a part of the user, such as a pointing gesture provided by a finger of the user pointing to the fourth area 416-4 of the user interface 430.

[0094] In some examples, as similarly discussed above, in response to detecting the gaze 421a directed toward the fourth region 416-4, the electronic device 401 generates suggested text for assisting the user in inputting text into the fourth region 416-4. Figure 4A , the electronic device 401 detects (e.g., using character recognition) the text label "allergens" in the fourth region 416-4 and uses information stored in a memory at the electronic device 401 or provided by an application (such as a health application) running on the electronic device 401 to identify information corresponding to known allergens for the user. In some examples, as similarly discussed above, the electronic device 401 generates suggested text based on previous text entered by the user into the user interface 430, such as the text 425-1 (e.g., Casey Williams) in the first region 416-1. Figure 4A As shown, electronic device 401 displays user interface element 440 including suggested text (e.g., "latex") corresponding to known allergens of a patient (e.g., a user of electronic device 401). In some examples, as similarly discussed above, electronic device 401 displays user interface element 440 at a location in three-dimensional environment 450 that is based on a location corresponding to fourth region 416-4 in user interface 430. For example, Figure 4A As shown and as similarly discussed above, electronic device 401 displays user interface element 440 above fourth area 416-4 in user interface 430 (e.g., at a location outside fourth area 416-4 in user interface 430). In some examples, user interface element 440 has the same FIG. 3A to FIG. 3M Describes one or more properties of a user interface element.

[0095] In some examples, such as Figure 4A As shown, the electronic device 401 alternatively detects the user's gaze 421b directed to the sixth area 416-6 in the three-dimensional environment 450. For example, the electronic device 401 detects the user's gaze 421b directed to the sixth area 416-6, instead of detecting the gaze 421a directed to the fourth area 416-4 as discussed above, or detects the gaze directed to the sixth area at a different time (e.g., before or after detecting the gaze 421a directed to the fourth area 416-4). In some examples, in response to detecting the gaze 421b directed to the sixth area 416-6, the electronic device 401 abandons displaying a user interface element for assisting the user in inputting text into the sixth area 416-6, such as Figure 4AFor example, as similarly discussed above, the electronic device 401 visually detects the text label "Family Medical History" in the sixth area 416-6 and initiates the location of information corresponding to the family medical history of the user (e.g., Casey Williams). Figure 4A , the electronic device 401 cannot generate suggested text based on the characteristics of the sixth area 416-6. For example, the electronic device 401 cannot locate information corresponding to the user's family medical history in the user data stored in the memory or in the user data provided by the application running on the electronic device 401. For another example, the electronic device 401 cannot generate suggested text for the sixth area 416-6 because the text label family medical history generates a large number of options (e.g., too many options to be displayed in a given user interface element in the three-dimensional environment 450) and / or because the topic is too broad to select one or more options for generating suggested text. In some examples, the electronic device 401 cannot generate suggested text for the sixth area 416-6 because the topic related to the user's family medical history is not applicable to the patient (e.g., because the patient (Casey Williams) does not have a known family history of certain diseases, cancers, mental health diseases, or other conditions).

[0096] Figure 4B An example is shown in which the electronic device 401 displays a three-dimensional environment 450 including a credit card machine 455 in the physical environment 418. For example, when displaying the three-dimensional environment 450 discussed above, the electronic device 401 visually detects the credit card machine 455 within the user's field of view in the portion of the physical environment 418 included in the three-dimensional environment 450 (e.g., at a checkout counter or other payment station). In some examples, as shown in FIG. Figure 4B As shown, the credit card machine 455 includes a display 456 and a PIN pad 458. Figure 4B As shown, three-dimensional environment 450 includes representations of a credit card machine 455 and a PIN pad 458, optionally as a computer-generated representation or as a pass-through representation.

[0097] exist Figure 4B In FIG. 4 , the user of the electronic device 401 is currently initiating payment for the product. Figure 4B As shown, a payment card 445 (e.g., a credit card, debit card, gift card, etc.) has been inserted into a credit card machine 455 for payment processing. Figure 4B , the credit card machine 455 is displaying a textual indication 457 on the display 456 prompting the user to enter the PIN associated with the payment card 445 to complete the payment process (eg, “Enter PIN”).

[0098] exist Figure 4B, when the credit card machine 455 is displaying a text indication 457 on the display 456, the electronic device 401 detects the user's gaze 421 directed toward the display 456 and / or the text indication 457 in the three-dimensional environment 450. Alternatively, in some examples, the electronic device 401 detects a gesture performed by a part of the user, such as a pointing gesture provided by a finger of the user directed toward the display 456 or the PIN pad 458.

[0099] In some examples, such as Figure 4B As shown, in response to detecting a user's gaze 421 directed toward display 456, electronic device 401 displays user interface element 442 in three-dimensional environment 450. In some examples, as similarly discussed herein, user interface element 442 includes suggested text, specifically a suggested PIN associated with payment card 445 (e.g., "5246"). In some examples, electronic device 401 generates suggested text in user interface element 442 based on one or more characteristics of an object currently visible in the user's field of view. For example, as discussed above, text indication 457 prompts the user to provide a PIN for payment card 445 currently inserted into credit card machine 455. In some examples, electronic device 401 is configured to visually detect (e.g., via one or more cameras of electronic device 401) payment card 445 in the user's field of view. Specifically, the electronic device 401 may detect (e.g., via object recognition, computer vision, or other image recognition techniques) at least a portion of the card number of the payment card 445, the visual appearance of the payment card 445 (e.g., color and / or design), the type of card, and / or the institution associated with the payment card 445 (e.g., a bank, credit institution, or other lender) to identify the specific card being used. Once the payment card 445 is identified, the electronic device 401 optionally locates information related to the payment card 445 stored in a memory at the electronic device 401, or via user data provided by one or more applications running on the electronic device 401 (such as a digital wallet application, a banking application, a note application, etc.). In some examples, such as Figure 4B As shown and as similarly discussed above, electronic device 401 displays user interface element 442 at a location in three-dimensional environment 450 that is based on a location corresponding to display 456 and / or PIN pad 458. For example, Figure 4B As shown, electronic device 401 displays user interface element 442 below display 456 and / or above PIN pad 458 to assist the user in selecting the appropriate button on PIN pad 458 to complete the transaction discussed above. In some examples, user interface element 442 has the above reference FIG. 3A to FIG. 3M Describes one or more properties of a user interface element.

[0100] Figure 4CAn example is shown in which electronic device 401 displays a three-dimensional environment 450 including an electronic door lock 461 in physical environment 418. For example, when displaying the three-dimensional environment 450 discussed above, electronic device 401 visually detects electronic door lock 461 in the user's field of view in a portion of physical environment 418 included in three-dimensional environment 450 (e.g., at the front door of a residence, apartment building, office, etc.). In some examples, such as Figure 4C As shown, the electronic door lock 461 includes a keypad 462 having a plurality of selectable keys. Figure 4C As shown, three-dimensional environment 450 includes a representation of an electronic door lock 461 and a keypad 462, optionally as a computer-generated representation or as a pass-through representation.

[0101] exist Figure 4C In FIG. 4 , the user of electronic device 401 is currently initiating unlocking of electronic door lock 461 to pass through door 460 in physical environment 418. For example, in Figure 4C , the door 460 is currently closed and the electronic door lock 461 is in a locked state, thereby requiring the user of the electronic device 401 to enter a specific code to unlock the electronic door lock 461.

[0102] exist Figure 4C , the electronic device 401 detects the user's gaze 421 in the three-dimensional environment 450, which is directed to the keypad 462 of the electronic door lock 461. Alternatively, in some examples, the electronic device 401 detects a gesture performed by a part of the user, such as a pointing gesture directed to the keypad 462 provided by the user's finger.

[0103] In some examples, such as Figure 4C As shown, in response to detecting the user's gaze 421 directed toward the keypad 462, the electronic device 401 displays the user interface element 443 in the three-dimensional environment 450. In some examples, such as Figure 4C As shown, user interface element 443 includes suggested text (e.g., "0101") corresponding to a suggested key code for unlocking electronic door lock 461. In some examples, electronic device 401 generates suggested text based on one or more characteristics of an object in the user's field of view. Figure 4CIn the example, the electronic device 401 visually detects the electronic door lock 461 in the user's field of view, including the visual appearance of the electronic door lock 461, the brand of the electronic door lock 461, the visual appearance of the door 460, and / or the visual appearance of the keypad 462. In addition, in some examples, the electronic device 401 determines the user's current location, which optionally corresponds to the location of the electronic device 401 (e.g., using GPS). Based on detecting the above environmental factors, the electronic device 401 optionally determines a specific electronic door lock in the user's field of view (e.g., the front door of the user's home to the front door of the user's workplace). As similarly discussed above, the electronic device 401 optionally locates information related to the key code of the electronic door lock 461 stored in the memory of the electronic device 401 or provided by one or more applications running on the electronic device 401. For example, the electronic device 401 has previously detected and captured an image (or data corresponding to the user stored in other ways) of a user entering a key code of the electronic door lock 461 (e.g., which causes the electronic door lock 461 to be unlocked). In some examples, the electronic device 401 identifies a key code based on user data provided by a note-taking application (e.g., that includes user-entered notes associated with passwords, key codes, and PINs), a messaging application (e.g., text or email), an application that facilitates stays and experiences (e.g., home rentals), or other applications.

[0104] In some examples, such as Figure 4C As shown and as similarly discussed above, electronic device 401 displays user interface element 443 at a location in three-dimensional environment 450 that is based on a location corresponding to keypad 462. For example, Figure 4C As shown, the electronic device 401 displays a user interface element 443 above the keypad 462 to assist the user in selecting the appropriate key on the keypad 462 to unlock the door 460. In some examples, the user interface element 443 has the above reference FIG. 3A to FIG. 3M Describes one or more properties of a user interface element.

[0105] It should be understood that the examples shown and described herein are merely exemplary, and additional and / or alternative elements related to displaying suggested content to the user may be provided in a three-dimensional environment. It should be understood that the appearance, shape, form, and size of each of the various user interface elements and objects shown and described herein are exemplary, and alternative appearances, shapes, forms, and / or sizes may be provided. For example, various user interface elements (e.g., user interface elements 320, 440, 442, and 443) including suggested text may be provided in alternative shapes (such as circular shapes, triangular shapes, etc.) other than rectangular shapes. Additionally or alternatively, in some examples, the various user interface elements described herein may be selected and / or manipulated via user input received by one or more independent input devices communicating with the electronic device. For example, where applicable, selection input (e.g., for initiating tracking of exercise activities) may be received via a physical input device (such as a mouse, a trackpad, a keyboard, etc.) communicating with the electronic device. In addition, in some examples, suggesting content to the user may include suggesting non-alphanumeric content. For example, if a particular area in a form or document presents a list of options (e.g., radio buttons or option buttons) from which a user is able to select one or more options that are applicable to the user, the electronic device may display one or more options in the list of options along with a visual indication (e.g., highlighting, underlining, circling, pointing arrows, etc.) indicating that the electronic device 301 is suggesting these options to the user in the manner discussed above.

[0106] Figure 5 5 is a flowchart illustrating an exemplary process for displaying a user interface element including suggested text based on detecting an area of ​​a physical object according to some examples of the present disclosure. In some examples, process 500 begins at an electronic device in communication with one or more displays, one or more input devices, and one or more cameras. In some examples, the electronic device is optionally similar to or corresponding to Figure 2 The head mounted display of the electronic device 201. Figure 5 As shown, in some examples, at 502, the electronic device captures one or more images of a physical environment including a first object via the one or more cameras. Figure 3A As shown, electronic device 301 detects document 315 in physical environment 318 , which is visible in three-dimensional environment 350 presented at electronic device 301 .

[0107] In some examples, at 504, based on detecting a first area of ​​a first object in the one or more images and satisfying one or more first criteria, the electronic device displays a first user interface element including a first suggested text in a computer-generated environment via the one or more displays, wherein the first user interface element is displayed at a first location based on the first area. For example, as referenced Figure 3B As described, the electronic device 301 detects the first area 316-1 of the document 315, and the user's gaze 321 points to the first area 316-1 in the three-dimensional environment 350. Figure 3B As shown, the electronic device 301 displays a user interface element 320-1 including suggested text (eg, "Casey Williams"), which is used to assist the user in entering text into the first area 316-1. Figure 3B As depicted, user interface element 320 - 1 is displayed in three-dimensional environment 350 at a location outside of first area 316 - 1 .

[0108] In some examples, at 506, based on detecting a second area of ​​the first object different from the first area in the one or more images and satisfying one or more second criteria, the electronic device displays a second user interface element including a second suggested text in a computer-generated environment via the one or more displays, wherein the second user interface element is displayed at a second location different from the first location based on the second area. For example, as referenced Figure 3E As described, the electronic device 301 detects the fourth area 316-4 of the document 315, and the finger of the user's hand 306 points to the fourth area 316-4 in the three-dimensional environment 350 (e.g., is touching the surface of the document 315 at the fourth area 316-4). Figure 3E As shown, the electronic device 301 displays a user interface element 320-3 including suggested text (eg, "Model A Brand"), which is used to assist the user in entering text into the fourth area 316-4. Figure 3E As depicted, user interface element 320 - 3 is displayed in three-dimensional environment 350 at a location outside of fourth region 316 - 4 .

[0109] It should be understood that process 500 is an example, and more, fewer, or different operations may be performed in the same or different order. In addition, the operations in the above process 500 are optionally performed by running an information processing device such as a general-purpose processor (e.g., as described in relation to Figure 2 The one or more functional modules in the dedicated chip and / or Figure 2 to be implemented by other components.

[0110] Therefore, according to the above, some examples of the present disclosure relate to a method, the method comprising: at an electronic device that communicates with one or more displays, one or more input devices, and one or more cameras: capturing one or more images of a physical environment including a first object via the one or more cameras; based on detecting a first area of ​​the first object in the one or more images and satisfying one or more first criteria, displaying a first user interface element including a first suggested text in a computer-generated environment via the one or more displays, wherein the first user interface element is displayed at a first position based on the first area; and based on detecting a second area of ​​the first object different from the first area in the one or more images and satisfying one or more second criteria, displaying a second user interface element including a second suggested text in the computer-generated environment via the one or more displays, wherein the second user interface element is displayed at a second position different from the first position based on the second area.

[0111] Additionally or alternatively, in some examples, the first object corresponds to a physical document. Additionally or alternatively, in some examples, the first object corresponds to a physical display that is displaying a user interface. Additionally or alternatively, in some examples, the first object corresponds to a locked device. Additionally or alternatively, in some examples, the first area corresponds to a first fillable field, and the second area corresponds to a second fillable field that is different from the first fillable field. Additionally or alternatively, in some examples, when a first area of ​​the first object is detected in the one or more images, the one or more first criteria include criteria that are satisfied when the user's gaze is directed to the first area, and when a second area of ​​the first object is detected in the one or more images, the one or more second criteria include criteria that are satisfied when the user's gaze is directed to the second area. Additionally or alternatively, in some examples, when a first area of ​​the first object is detected in the one or more images, the one or more first criteria include criteria that are satisfied when a corresponding object associated with the user is detected within a threshold distance of the first area, and when a second area of ​​the first object is detected in the one or more images, the one or more second criteria include criteria that are satisfied when a corresponding object associated with the user is detected within a threshold distance of the second area. Additionally or alternatively, in some examples, the one or more first criteria include criteria that are satisfied when the first suggested text is associated with a threshold confidence level based on one or more characteristics of the first area, and the one or more second criteria include criteria that are satisfied when the second suggested text is associated with a threshold confidence level based on one or more characteristics of the second area.

[0112] Additionally or alternatively, in some examples, the one or more first criteria include criteria that are satisfied when the first content included in the first area is applicable to the user, and the one or more second criteria include criteria that are satisfied when the second content included in the second area is applicable to the user. Additionally or alternatively, in some examples, when a first area of ​​a first object is detected in the one or more images, the one or more first criteria include criteria that are satisfied when the first object remains in the field of view of the electronic device for at least a threshold amount of time, and when a second area of ​​the first object is detected in the one or more images, the one or more second criteria include criteria that are satisfied when the second object remains in the field of view for at least a threshold amount of time. Additionally or alternatively, in some examples, the first suggestion text is displayed based on one or more features of the first area, and the second suggestion text is displayed based on one or more features of the second area. Additionally or alternatively, in some examples, the one or more features of the first area include the first text included in the first area, and the one or more features of the second area include the second text included in the second area. Additionally or alternatively, in some examples, the one or more features of the first area and the one or more features of the second area include the type of the first object. Additionally or alternatively, in some examples, the first suggestion text is displayed based on first user data accessible on the electronic device, and the second suggestion text is displayed based on second user data accessible on the electronic device. Additionally or alternatively, in some examples, the first user data and the second user data are stored in a memory of the electronic device. Additionally or alternatively, in some examples, the first user data and the second user data are provided by one or more applications running on the electronic device. Additionally or alternatively, in some examples, the first user data and the second user data can be accessed based on the current location of the electronic device.

[0113] Additionally or alternatively, in some examples, displaying the first user interface element includes: based on determining that the first suggestion text includes multiple first options, displaying the multiple first options as a first list in the first user interface element, and displaying the second user interface element includes: based on determining that the second suggestion text includes multiple second options, displaying the multiple second options as a second list in the second user interface element. Additionally or alternatively, in some examples, the first position is outside the boundary of the first area, and the second position is outside the boundary of the second area. Additionally or alternatively, in some examples, the first position is at least a threshold distance from the first area, and the second position is at least a threshold distance from the second area. Additionally or alternatively, in some examples, the method further includes: based on detecting a third area of ​​the first object in the one or more images that is different from the first area and the second area and satisfying one or more third criteria, displaying a third user interface element including a third suggestion text in a computer-generated environment via the one or more displays, wherein the third user interface element is displayed at a third position different from the first position and the second position based on the third area. Additionally or alternatively, in some examples, the method further comprises: based on determining that the one or more first criteria and the one or more second criteria are satisfied, simultaneously displaying the first user interface element at the first location and the second user interface element at the second location in the computer-generated environment via the one or more displays. Additionally or alternatively, in some examples, the method further comprises:

[0114] After displaying a first user interface element at a first position in a computer-generated environment, detecting first text in a first area of ​​a first object, wherein the first text had not been detected before displaying the first user interface element; and displaying a third user interface element including third suggested text in the computer-generated environment via the one or more displays based on detecting a third area of ​​the first object in the one or more images and satisfying one or more third criteria, wherein the third user interface element is displayed at a third position based on the first area and the third suggested text is based on the first text.

[0115] Additionally or alternatively, in some examples, the method further includes: while displaying the first user interface element at the first location in the computer-generated environment, detecting first text in a first area of ​​the first object, wherein the first text had not been detected before the first user interface element was displayed; and in response to detecting the first text in the first area of ​​the first object, stopping displaying the first user interface element in the computer-generated environment. Additionally or alternatively, in some examples, the method further includes: after displaying the first user interface element at the first location in the computer-generated environment, detecting the first text in the first area of ​​the first object, wherein the first text had not been detected before the first user interface element was displayed; and in response to detecting the first text, displaying a visual indication in the computer-generated environment via the one or more displays based on determining that at least a portion of the first text is different from a corresponding portion of the first suggested text. Additionally or alternatively, in some examples, the visual indication is displayed together with the at least a portion of the first text. Additionally or alternatively, in some examples, after detecting the first text in the first area, the visual indication is displayed for a threshold amount of time. Additionally or alternatively, in some examples, the method also includes: based on detecting a first area of ​​a first object in the one or more images and not satisfying the one or more first criteria, abandoning displaying a first user interface element including first suggested text in a computer-generated environment via the one or more displays; and based on detecting a second area of ​​the first object in the one or more images and not satisfying the one or more second criteria, abandoning displaying a second user interface element including second suggested text in the computer-generated environment.

[0116] Additionally or alternatively, in some examples, when a first area of ​​a first object is detected in the one or more images, the one or more first criteria include criteria that are satisfied when a corresponding object associated with the user is detected as pointing to the first area for at least a threshold amount of time, and when a second area of ​​the first object is detected in the one or more images, the one or more second criteria include criteria that are satisfied when the corresponding object associated with the user is detected as pointing to the second area for at least a threshold amount of time. Additionally or alternatively, in some examples, the method further includes: based on detecting a corresponding object associated with the user pointing to the first area or the second area of ​​the first object, displaying a visual indication of the passage of a threshold amount of time in a computer-generated environment via the one or more displays. Additionally or alternatively, in some examples, the method further includes: capturing one or more second images of the physical environment via the one or more cameras while a visual indication of the lapse of a threshold amount of time is displayed in the computer-generated environment; detecting in the one or more second images that a corresponding object associated with the user no longer points to the first area or the second area of ​​the first object; and in response to detecting that the corresponding object associated with the user no longer points to the first area or the second area, stopping displaying the visual indication in the computer-generated environment and abandoning displaying the first user interface element or the second user interface element in the computer-generated environment based on determining that the threshold amount of time has not elapsed before detecting that the corresponding object associated with the user points to the first area or the second area.

[0117] Some examples of the present disclosure relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for executing any of the above methods.

[0118] Some examples of the present disclosure relate to a non-transitory computer-readable storage medium that stores one or more programs that include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the above methods.

[0119] Some examples of the present disclosure relate to an electronic device comprising one or more processors, a memory, and a component for performing any of the above methods.

[0120] Some examples of the present disclosure relate to an information processing apparatus used in an electronic device, the information processing apparatus including a component for executing any one of the above methods.

[0121] The present disclosure contemplates that, in some instances, the data utilized may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, content consumption activity, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information. Specifically, as described herein, one aspect of the present disclosure is tracking a user's biometric data.

[0122] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to display suggestion text that changes based on changes in the user's biometric data. For example, the suggestion text is updated based on changes in the user's age, height, weight, and / or medical history.

[0123] The disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information data will comply with sound privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easy for users to access and should be updated as changes in the collection and / or use of data occur. Personal information from users should be collected for legal and reasonable entity purposes and should not be shared or sold outside these legal purposes. In addition, such collection / sharing should be carried out after receiving the informed consent of the user. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data, and ensure that other entities with access to personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adapted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including considerations of special jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.

[0124] Regardless of the foregoing, the present disclosure also contemplates examples where users selectively block the use or access of personal information data. That is, the present disclosure contemplates hardware elements and / or software elements to prevent or block access to such personal information data. For example, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" to participate in the collection of personal information data during or at any time after registration for a service. In another example, a user may choose not to enable the recording of personal information data in a particular application (e.g., a first application and / or a second application). In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed after the collection is initiated, and then the user may be reminded again just before the device accesses the personal information data.

[0125] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting data collection and data deletion. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0126] For the purpose of explanation, the foregoing description is described by reference to specific examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Based on the above teachings, many modifications and variations are possible. The examples are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so as to enable other technical personnel in the field to best use the present disclosure and the various described examples with various modifications suitable for the specific purpose contemplated.

Claims

1. A method, comprising: At an electronic device in communication with one or more displays, one or more input devices, and one or more cameras: capturing, via the one or more cameras, one or more images of a physical environment including a first object; displaying, via the one or more displays, a first user interface element including first suggested text in a computer-generated environment based on detecting a first area of ​​the first object in the one or more images and satisfying one or more first criteria, wherein the first user interface element is displayed at a first location based on the first area; as well as Based on detecting a second area of ​​the first object that is different from the first area in the one or more images and satisfying one or more second criteria, displaying a second user interface element including second suggested text in the computer-generated environment via the one or more displays, wherein the second user interface element is displayed at a second position that is different from the first position based on the second area.

2. The method according to claim 1, wherein: The first area corresponds to a first fillable field; and The second area corresponds to a second fillable field different from the first fillable field.

3. The method according to claim 1, wherein: When the first area of ​​the first object is detected in the one or more images, the one or more first criteria include a criterion that is satisfied when the user's gaze is directed toward the first area; and When the second area of ​​the first object is detected in the one or more images, the one or more second criteria include a criterion that is satisfied when the gaze of the user is directed toward the second area.

4. The method according to claim 1, wherein: When the first region of the first object is detected in the one or more images, the one or more first criteria include a criterion that is satisfied when a corresponding object associated with the user is detected within a threshold distance of the first region; and When the second region of the first object is detected in the one or more images, the one or more second criteria include a criterion that is satisfied when the corresponding object associated with the user is detected within the threshold distance of the second region.

5. The method according to claim 1, wherein: When the first region of the first object is detected in the one or more images, the one or more first criteria include a criterion that is satisfied when a corresponding object associated with the user is detected as being directed toward the first region for at least a threshold amount of time; and When the second region of the first object is detected in the one or more images, the one or more second criteria include a criterion that is satisfied when the corresponding object associated with the user is detected as pointing toward the second region for at least the threshold amount of time.

6. The method according to claim 1, wherein: The one or more first criteria include a criterion that is satisfied when the first suggested text is associated with a threshold confidence level based on one or more characteristics of the first region; and The one or more second criteria include a criterion that is satisfied when the second suggested text is associated with the threshold confidence level based on one or more characteristics of the second region.

7. The method according to claim 1, wherein: When the first area of ​​the first object is detected in the one or more images, the one or more first criteria include a criterion that is satisfied when the first object remains in a field of view of the electronic device for at least a threshold amount of time; and When the second area of ​​the first object is detected in the one or more images, the one or more second criteria include a criterion that is satisfied when the first object remains in the field of view for at least the threshold amount of time.

8. The method according to claim 1, wherein: displaying the first suggested text based on one or more characteristics of the first area; and The second suggested text is displayed based on one or more characteristics of the second area.

9. The method according to claim 8, wherein: The one or more features of the first region include first text included in the first region; and The one or more features of the second region include second text included in the second region.

10. The method of claim 1, wherein: displaying the first suggested text based on first user data accessible on the electronic device; and The second suggestion text is displayed based on second user data accessible on the electronic device.

11. The method according to claim 1, wherein: Displaying the first user interface element includes: based on determining that the first suggestion text includes a plurality of first options, displaying the plurality of first options as a first list in the first user interface element; and Displaying the second user interface element includes: based on determining that the second suggestion text includes multiple second options, displaying the multiple second options as a second list in the second user interface element.

12. The method of claim 1, wherein: The first location is outside the boundaries of the first area; and The second location is outside the boundary of the second area.

13. The method of claim 1, wherein: The first location is at least a threshold distance from the first area; and The second position is at least the threshold distance from the second area.

14. The method according to claim 1, further comprising: Based on detecting a third area of ​​the first object that is different from the first area and the second area in the one or more images and satisfying one or more third criteria, displaying a third user interface element including a third suggested text in the computer-generated environment via the one or more displays, wherein the third user interface element is displayed at a third position that is different from the first position and the second position based on the third area.

15. The method according to claim 1, further comprising: Based on determining that the one or more first criteria and the one or more second criteria are satisfied, a first user interface element is simultaneously displayed at the first location and the second user interface element is simultaneously displayed at the second location in the computer-generated environment via the one or more displays.

16. The method according to claim 1, further comprising: detecting first text in the first region of the first object after displaying the first user interface element at the first location in the computer-generated environment, wherein the first text was not detected before displaying the first user interface element; as well as Based on detecting a third area of ​​the first object in the one or more images and satisfying one or more third criteria, displaying a third user interface element including third suggested text in the computer-generated environment via the one or more displays, wherein the third user interface element is displayed at a third position based on the first area and the third suggested text is based on the first text.

17. The method according to claim 1, further comprising: detecting first text in the first region of the first object while displaying the first user interface element at the first location in the computer-generated environment, wherein the first text was not detected prior to displaying the first user interface element; as well as In response to detecting the first text in the first area of ​​the first object, ceasing to display the first user interface element in the computer-generated environment.

18. The method according to claim 1, further comprising: detecting first text in the first region of the first object after displaying the first user interface element at the first location in the computer-generated environment, wherein the first text was not detected before displaying the first user interface element; as well as In response to detecting the first text: Based on determining that at least a portion of the first text is different from a corresponding portion of the first suggested text, a visual indication is displayed in the computer-generated environment via the one or more displays.

19. An electronic device, comprising: one or more processors; Memory; and One or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any one of the methods according to claims 1 to 18.

20. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by one or more processors of an electronic device, cause the electronic device to perform any one of the methods according to claims 1 to 18.

Images