Method and apparatus for resolving focus conflicts

Abstract

The invention relates to a method and device for resolving focus conflicts. In one implementation, a method of resolving focus collisions in a computer generated reality (CGR) environment is performed by a device including a processor, a non-transitory memory, an image sensor, and a display. The method includes capturing, using an image sensor, an image of a scene including a real object at a first distance from the device in a particular direction. The method includes displaying a CGR environment on a display, the CGR environment including a virtual object at a second distance from the device in a particular direction. In accordance with a determination that the second distance is less than the first distance, the CGR environment includes a virtual object overlying the scene. In accordance with a determination that the second distance is greater than the first distance, the CGR environment includes a virtual object having an obfuscation zone that obfuscates a real object within at least a portion of the obfuscation zone.

Description

[0001] Cross - reference to related applications

[0002] This application is a divisional application of the Chinese patent application with the national application number 202010721303.8, the application date of July 24, 2020, and the invention title of "Method and Device for Solving Focus Conflict". Technical Field

[0003] The present disclosure generally relates to systems, methods, and devices for solving focus conflicts between real objects and virtual objects. Background Art

[0004] The physical environment refers to the physical world that people can sense and / or interact with without the help of an electronic system. Physical environments such as physical parks include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0005] In contrast, a computer-generated reality (CGR) environment refers to a fully or partially simulated environment that people perceive and / or interact with via an electronic system. In CGR, a subset of a person's physical movements or their representations are tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one physical law. For example, a CGR system can detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in the physical environment. In some cases (e.g., for accessibility reasons), the adjustment of the characteristics of virtual objects in the CGR environment can be made in response to a representation of a physical movement (e.g., a voice command).

[0006] A person can use any of their senses to sense and / or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of point audio sources in 3D space. Also, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In certain CGR environments, a person can sense and / or interact only with audio objects.

[0007] Examples of CGR include virtual reality and mixed reality.

[0008] A virtual reality (VR) environment is an artificial environment that is designed to be based entirely on computer-generated sensory inputs for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with the virtual objects in the VR environment by way of a simulation of the person's presence within the computer-generated environment and / or by way of a simulation of a subset of the person's physical movements within the computer-generated environment.

[0009] In contrast to a VR environment, which is designed to be based entirely on computer-generated sensory inputs, a mixed reality (MR) environment is an artificial environment that is designed to incorporate sensory inputs or representations thereof from a physical environment in addition to including computer-generated sensory inputs (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any condition between a fully physical environment at one end and a virtual reality environment at the other end, but excluding these two ends.

[0010] In some MR environments, the computer-generated sensory inputs can respond to changes in the sensory inputs from the physical environment. Additionally, some electronic systems for presenting an MR environment can track the position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement such that a virtual tree appears stationary relative to the physical ground.

[0011] Examples of mixed reality include augmented reality and augmented virtuality.

[0012] An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are superimposed on a physical environment or its representation. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or translucent display such that a person using the system perceives the virtual objects superimposed on the physical environment. Alternatively, the system can have an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment. The system combines the images or video with the virtual objects and presents the combination on the opaque display. A person using the system indirectly views the physical environment via the images or video of the physical environment and perceives the virtual objects superimposed on the physical environment. As used herein, the video of the physical environment displayed on the opaque display is referred to as “passthrough video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Further alternatively, the system can have a projection system that projects virtual objects into the physical environment, such as as a hologram or on a physical surface, such that a person using the system perceives the virtual objects superimposed on the physical environment.

[0013] An augmented reality environment is also a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing passthrough video, the system can transform one or more sensor images to impose an alternative perspective (e.g., viewpoint) different from the perspective captured by the imaging sensors. As another example, a representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions thereof such that the modified portions can be a representative but not a true version of the originally captured image. As yet another example, a representation of the physical environment can be transformed by graphically removing portions thereof or blurring portions thereof.

[0014] An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but a person's face is realistically reproduced from an image of a physical person. As another example, a virtual object can assume the shape or color of a physical item imaged by one or more imaging sensors. As yet another example, a virtual object can assume a shadow that conforms to the position of the sun in the physical environment.

[0015] There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed to be placed on a person's eye (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to receive an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system can have a transparent or translucent display instead of an opaque display. The transparent or translucent display can have a medium through which light representing an image is directed to a person's eye. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one particular implementation, the transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection techniques that project graphical images onto a person's retina. The projection system can also be configured to project virtual objects into the physical environment, such as as a hologram or on a physical surface.

[0016] In various particular implementations, the CGR environment includes one or more real objects and one or more virtual objects. In various particular implementations, virtual objects are rendered at a distance behind real objects such that they are not occluded by the real objects. This creates a focus conflict where the user sees the virtual object and obtains depth cues as if the virtual object were farther away than the real object and should thus be occluded by the physical object, but is not. It may be desirable to effectively resolve such focus conflicts. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Accordingly, the present disclosure may be understood by those of ordinary skill in the art, and a more detailed description may be found by reference to aspects of some illustrative particular implementations, some of which are shown in the drawings.

[0018] Figure 1 is a block diagram of an exemplary operating environment according to some particular implementations.

[0019] Figure 2 is a block diagram of an exemplary controller according to some particular implementations.

[0020] Figure 3 is a block diagram of an exemplary electronic device according to some specific implementations.

[0021] Figures 4A to 4G illustrates a CGR environment based on a real environment surveyed by a scene camera of a device according to various specific implementations.

[0022] Figure 5 is a flowchart representation of a method for resolving focus conflicts according to some specific implementations.

[0023] According to common practice, the various feature portions shown in the drawings may not be drawn to scale. Thus, for clarity, the sizes of the various feature portions may be arbitrarily enlarged or reduced. Additionally, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and the drawings, like reference numerals may be used to represent like feature portions. SUMMARY OF THE INVENTION

[0024] The various specific implementations disclosed herein include devices, systems, and methods for resolving focus conflicts in a computer-generated reality (CGR) environment. In various specific implementations, the method is performed at a device including a processor, a non-transitory memory, an image sensor, and a display. The method includes capturing, using the image sensor, an image of a scene including a real object at a first distance in a particular direction. The method includes displaying, on the display, a computer-generated reality (CGR) environment that includes a virtual object at a second distance from the device in the particular direction. Based on determining that the second distance is less than the first distance, the CGR environment includes the virtual object overlaid on the scene. Based on determining that the second distance is greater than the first distance, the CGR environment includes the virtual object with a confusion zone.

[0025] According to some specific implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors. The one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some specific implementations, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of a device, cause the device to perform or cause to perform any of the methods described herein. According to some specific implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing to perform any of the methods described herein. DETAILED DESCRIPTION

[0026] Numerous details are described to provide a thorough understanding of the example embodiments shown in the drawings. However, the drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. Those of ordinary skill in the art will understand that other effective aspects and / or variations do not include all of the specific details described herein. Additionally, well-known systems, methods, components, devices, and circuits have not been described in detail so as not to obscure more relevant aspects of the exemplary embodiments described herein.

[0027] Various CGR environments include both real objects and virtual objects. When rendering a virtual object at a distance farther than the distance to a real object, there is a focus conflict. A user may be confused or perplexed by seeing a virtual object that should be occluded by a real object. For example, while a user is sitting on a bus or an airplane, there is a short distance between the user and the seat in front of them. However, the user may wish to view virtual content at a distance greater than that short distance without a focus conflict. Similarly, while sitting at a short distance from a table with a computer monitor, the user may wish to view virtual content at a distance greater than that short distance without a focus conflict. Thus, in various embodiments, the virtual content is displayed at a greater distance in the case of resolving the focus conflict by means of a confounding zone (e.g., surrounding the virtual content) having the virtual content. The confounding zone blurs, dims, and / or occludes the portion of the real object within the confounding zone.

[0028] As another example, in various embodiments, a CGR environment includes a first real environment in which a user is located and a head portrait of a person in a second real environment that represents being away from the first real environment. Thus, the CGR environment allows for interaction between the user and the person (through the head portrait). When the person moves within the second real environment, the head portrait moves correspondingly within the CGR environment. In various embodiments, the second real environment may be larger than the first real environment, and as the person moves within the second real environment, the distance by which the head portrait moves is greater than the distance to the wall of the first real environment. The CGR environment displays the head portrait with a confounding zone (e.g., surrounding the head portrait) without occluding the head portrait and hindering the interaction between the user and the person.

[0029] Figure 1 is a block diagram of an exemplary operating environment 100 according to some embodiments. Although relevant features are shown, those of ordinary skill in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary embodiments disclosed herein. To that end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120.

[0030] In some embodiments, the controller 110 is configured to manage and coordinate the user's CGR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Reference is made below toFigure 2 Controller 110 is described in more detail. In some specific embodiments, controller 110 is a computing device that is local or remote relative to scene 105. For example, controller 110 is a local server located within scene 105. As another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside of scene 105. In some specific embodiments, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). As another example, controller 110 is included within the housing of electronic device 120. In some specific embodiments, the functionality of controller 110 is provided by and / or in combination with electronic device 120.

[0031] In some specific embodiments, electronic device 120 is configured to provide a CGR experience to a user. In some specific embodiments, electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some specific embodiments, while the user is physically present within scene 105, electronic device 120 presents CGR content to the user via display 122, the scene including table 107 within the field of view 111 of electronic device 120. In some specific embodiments, the user holds electronic device 120 in one or both of his / her hands. In some specific embodiments, while providing augmented reality (AR) content, electronic device 120 is configured to display AR objects (e.g., AR cylinder 109) and effect video passthrough of scene 105 (e.g., including representation 117 of table 107) on display 122. Reference is made below to Figure 3 Electronic device 120 is described in more detail.

[0032] According to some specific embodiments, while the user is virtually and / or physically present within scene 105, electronic device 120 provides a CGR experience to the user.

[0033] In some specific implementations, the user wears the electronic device 120 on his / her head. For example, in some specific implementations, the electronic device includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted enclosure (HME). Accordingly, the electronic device 120 includes one or more CGR displays configured to display CGR content. For example, in various specific implementations, the electronic device 120 surrounds the user's field of view. In some specific implementations, the electronic device 120 is a handheld device (such as a smart phone or a tablet computer) configured to present CGR content, and the user no longer wears the electronic device 120 but holds the device while orienting the display towards the user's field of view and the camera towards the scene 105. In some specific implementations, the handheld device can be placed inside an enclosure that can be worn on the user's head. In some specific implementations, the electronic device 120 is replaced by a CGR pod, enclosure, or chamber configured to present CGR content, in which the user no longer wears or holds the electronic device 120.

[0034] Figure 2 FIG. is a block diagram of an example of the controller 110 according to some specific implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the specific implementations disclosed herein. To that end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 202 (e.g., microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, global system for mobile communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0035] In some specific implementations, the one or more communication buses 204 include circuitry for interconnecting system components and controlling communication between system components. In some specific implementations, one or more of the I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0036] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disc storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes non-transitory computer-readable storage medium. In some specific implementations, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores the following programs, modules, and data structures, or subsets thereof, which include an optional operating system 230 and a CGR content module 240.

[0037] The operating system 230 includes procedures for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the CGR content module 240 is configured to manage and coordinate the presentation of CGR content for one or more users (e.g., a single set of CGR content for one or more users, or multiple sets of CGR content for corresponding groups of one or more users). To this end, in various specific implementations, the CGR content module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

[0038] In some specific implementations, the data acquisition unit 242 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least Figure 1 the electronic device 120. To this end, in various specific implementations, the data acquisition unit 242 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0039] In some specific implementations, the tracking unit 244 is configured to map the scene 105 and at least track the position / orientation of the electronic device 120 relative to Figure 1 the scene 105. To this end, in various specific implementations, the tracking unit 244 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0040] In some specific implementations, the coordination unit 246 is configured to manage and coordinate the CGR content presented to the user by the electronic device 120. To this end, in various specific implementations, the coordination unit 246 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0041] In some specific implementations, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the electronic device 120. To this end, in various specific implementations, the data transmission unit 248 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0042] Although the data acquisition unit 242, the tracking unit 244, the coordination unit 246, and the data transmission unit 248 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 242, the tracking unit 244, the coordination unit 246, and the data transmission unit 248 may be located in separate computing devices.

[0043] In addition, Figure 2 It is more used as a functional description of various features that may exist in a particular implementation, different from the structural schematic diagram of the specific implementation described herein. As will be recognized by those of ordinary skill in the art, the items shown separately can be combined, and some items can be separated. For example, Figure 2 Some of the functional modules shown separately in can be implemented in a single module, and the various functions of a single functional block can be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the specific division of functions, as well as how the features are allocated therein, will vary according to the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware selected for the particular implementation.

[0044] Figure 3is a block diagram of an example of an electronic device 120 according to some specific implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the specific implementations disclosed herein. To that end, as a non-limiting example, in some specific implementations, the electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional internal and / or external facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0045] In some specific implementations, one or more communication buses 304 include circuitry for interconnecting and controlling communication between system components. In some specific implementations, one or more I / O devices and sensors 306 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more microphones, one or more speakers, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a respiration monitor, a galvanic skin monitor, a blood oxygen sensor, a blood glucose sensor, etc.), a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), etc.

[0046] In some specific implementations, one or more CGR displays 312 are configured to display CGR content to a user. In some specific implementations, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some specific implementations, one or more CGR displays 312 correspond to diffractive, reflective, polarization, holographic, and other waveguide displays. For example, the electronic device 120 includes a single CGR display. As another example, the electronic device 120 includes a CGR display for each eye of the user. In some specific implementations, one or more CGR displays 312 are capable of presenting MR and VR content.

[0047] In some specific implementations, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face (including the user's eyes) (and thus can be referred to as an eye-tracking camera). In some specific implementations, one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to the scene that the user would see when the electronic device 120 is not present (and thus can be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0048] The memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some specific implementations, the memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. The memory 320 includes non-transitory computer-readable storage media. In some specific implementations, the memory 320 or the non-transitory computer-readable storage media of the memory 320 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 330 and a CGR rendering module 340.

[0049] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the CGR rendering module 340 is configured to present CGR content to a user via one or more CGR displays 312 and / or I / O devices and sensors 306 (such as one or more speakers). To this end, in various specific implementations, the CGR rendering module 340 includes a data acquisition unit 342, a focus conflict unit 344, a CGR content presentation unit 346, and a data transmission unit 348.

[0050] In some specific implementations, the data acquisition unit 342 is configured to acquire data (such as rendering data, interaction data, sensor data, location data, etc.) at least from Figure 1 the controller 110. In various specific implementations, the data acquisition unit 342 is configured to acquire data from the I / O devices and sensors 306. To this end, in various specific implementations, the data acquisition unit 342 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0051] In some specific implementations, the focus conflict unit 344 is configured to detect and resolve focus conflicts in the CGR environment. To this end, in various specific implementations, the focus conflict unit 344 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0052] In some specific implementations, the CGR content presentation unit 346 is configured to present CGR content to a user. In various specific implementations, the CGR content presentation unit 346 controls one or more CGR displays 312 to display a confusion zone around a virtual object at a distance farther than a real object. To this end, in various specific implementations, the CGR content presentation unit 346 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0053] In some specific implementations, the data transmission unit 348 is configured to transmit data (such as rendering data, location data, etc.) at least to the controller 110. To this end, in various specific implementations, the data transmission unit 348 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0054] Although the data acquisition unit 342, the focus conflict unit 344, the CGR content presentation unit 346, and the data transmission unit 348 are shown as being located on a single device (such as Figure 1 the electronic device 120), it should be understood that in other specific implementations, any combination of the data acquisition unit 342, the focus conflict unit 344, the CGR content presentation unit 346, and the data transmission unit 348 may be located in separate computing devices.

[0055] In addition, Figure 3 it is more used as a functional description of various features that may exist in a particular implementation, different from the structural schematic diagrams of the specific implementations described herein. As will be recognized by those of ordinary skill in the art, items shown separately can be combined, and some items can be separated. For example, Figure 3 some of the functional modules shown separately in can be implemented in a single module, and the various functions of a single functional block can be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the specific division of functions and how to allocate features therein will vary according to the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware selected for the particular implementation.

[0056] Figure 4A A CGR environment 400 based on a real environment surveyed by a scene camera of a device at a first time is shown. In various specific implementations, the scene camera is part of a device worn by a user and includes a display that displays the first CGR environment 400. Thus, in various specific implementations, the user is physically present in the environment. In various specific implementations, the scene camera is part of a remote device (such as a drone or a robotic avatar) that transmits images from the scene camera to a local device worn by the user and includes a display that displays the CGR environment 400.

[0057] The CGR environment 400 includes a plurality of objects, including one or more real objects (e.g., table 412, television 413, lamp 414, wall 416, and floor 417) and one or more virtual objects (e.g., avatar 422). In various specific implementations, each object is displayed at a location in the first CGR environment 400 (e.g., at a location defined by three coordinates in a three-dimensional (3D) CGR coordinate system). Thus, when the user moves in the CGR environment 400 (e.g., changes position and / or orientation), the objects move on the display of the electronic device but maintain their positions in the CGR environment 400. In various specific implementations, some virtual objects are displayed at locations on the display such that when the user moves in the CGR environment 400, the objects are stationary on the display of the electronic device.

[0058] In various specific implementations, the avatar 422 represents a person away from the real environment (e.g., in a second real environment). When the person moves within the second real environment, the avatar 422 moves correspondingly in the CGR environment 400.

[0059] At a first time, an avatar 422 is displayed at a first position in front of a wall 416. The distance from a particular direction of the scene camera to the avatar 422 is less than the distance from the particular direction of the scene camera to the wall 416. Thus, there is no focus conflict.

[0060] Figure 4B Illustrated is the Figure 4A CGR environment 400 at a second time. At the second time, the avatar 422 is displayed at a second position closer to the wall 416 (and the television 413) but still in front of the wall (and the television). In various embodiments, the avatar 422 moves in response to the person represented by the avatar 422 moving within the second real environment. The distance from a particular direction of the scene camera to the avatar 422 is greater than Figure 4A the distance in Figure 4B Thus, the avatar 422 as shown in Figure 4A is smaller than the avatar 422 as shown in

[0061] Figure 4C Illustrated is the Figure 4A CGR environment 400 at a third time. At the third time, the avatar 422 is displayed at a third position further away from the scene camera and behind the wall 416 (and the television 413). In various embodiments, the avatar 422 moves in response to the person represented by the avatar 422 moving within the second real environment. The distance from a particular direction of the scene camera to the avatar 422 is greater than Figure 4B the distance in Figure 4C Thus, the avatar 422 as shown in Figure 4B is smaller than the avatar 422 as shown in

[0062] The user receives a depth cue that indicates that the avatar 422 is at a particular distance further away than the distance to the wall 416. For example, the user can determine the distance to the avatar 422 through parallax derivation based on different views of the two eyes or based on different views obtained by moving within the real environment. The user can determine the distance to the avatar 422 by noticing that the size of the avatar 422 has shrunk as it moves from the first position to the second position to the third position.

[0063] Because the distance to the avatar 422 is greater than the distance to the wall 416, if the avatar 422 were a real object, the avatar would be occluded by the wall 416. The user may be confused or puzzled that the avatar 422 should be occluded but is clearly visible.

[0064] Figure 4D The CGR environment 400 at the third time is shown with a first focus conflict resolution. At the third time, an avatar 422 is shown at a third position behind the wall 416. In response to determining that the distance to the avatar 422 is greater than the distance to the wall 416, the CGR environment 400 includes a masking area 431 with the avatar 422, which partially obscures the wall 416 and the television 413. In Figure 4A it, the masking area 431 is a white oval surrounding the avatar 422. In various embodiments, the masking area 431 is white, black, or any other color. In various embodiments, the masking area 431 is oval, rectangular, or any other shape. In various embodiments, the masking area 431 has the same shape as but is larger than the avatar 422, creating a masking halo around the avatar 422. In various embodiments, the size of the masking area 431 is proportional to the size of the avatar 422 (and the size of the masking area is larger than the size of the avatar). For example, in various embodiments, the masking area 431 is 1.25 times larger, 1.5 times larger, 2 times larger, or 3 times larger than the avatar 422 (in any region or any specific dimension). Figure 4D

[0065] Figure 4E The CGR environment 400 at the third time is shown with a second focus conflict resolution. At the third time, an avatar 422 is shown at a third position behind the wall 416. In response to determining that the distance to the avatar 422 is greater than the distance to the wall 416, the CGR environment 400 includes a blur area 432 with the avatar 422, which blurs the wall 416 and the television 413 partially. Thus, in the area adjacent to the avatar 422, the wall 416 and the television 413 are blurred. However, in various embodiments, the avatar 422 is not blurred. Figure 4A

[0066] In Figure 4E it, the blur area 432 is an oval surrounding the avatar 422. In various embodiments, the blur area 432 is oval, rectangular, or any other shape. In various embodiments, the blur area 432 has the same shape as but is larger than the avatar 422, creating a blur halo around the avatar 422. In various embodiments, the size of the blur area 432 is proportional to the size of the avatar 422 (and the size of the blur area is larger than the size of the avatar). For example, in various embodiments, the blur area 432 is 1.25 times larger, 1.5 times larger, 2 times larger, or 3 times larger than the avatar 422 (in any region or any specific dimension). In various embodiments, the blur area 432 occupies the entire CGR environment 400 (except for the avatar 422). ​​

[0067] In various embodiments, the blurred region 432 is also a dimming region that dims the region adjacent to the avatar 422 such that the wall 416 and the television 413 are blurred and dimmed in the region surrounding the avatar 422. In various embodiments, the avatar 422 is neither blurred nor dimmed.

[0068] In various embodiments, the blurred region 432 is replaced with a dimming region having the avatar 422 such that the wall 416 and the television 413 are dimmed but not blurred in the region adjacent to the avatar 422. In various embodiments, the avatar 422 is not dimmed.

[0069] Figure 4F The CGR environment 400 at the third time is shown with a third focus conflict resolution. Figure 4A At the third time, the avatar 422 is displayed at a third position behind the wall 416. In response to determining that the distance to the avatar 422 is greater than the distance to the wall 416, the CGR environment 400 includes a portal region 433 having the avatar 422 that partially occludes the wall 416 and the television 413. Accordingly, the wall 416 and the television 413 cannot be seen in the region adjacent to the avatar 422. Instead, in the portal region 433, a virtual world including the avatar 422 is displayed. In various embodiments, the virtual world includes a virtual floor 435 in the same plane as the real floor 417. In various embodiments, the virtual floor 435 is displayed as a grid. In various embodiments, a virtual shadow of the avatar 422 is displayed on the virtual floor 435.

[0070] In Figure 4F the portal region 433 is an ellipse surrounding the avatar 422. In various embodiments, the portal region 433 is an ellipse, a rectangle, or any other shape. In various embodiments, the portal region 433 includes a halo effect at the outer edge of the portal region 433. In various embodiments, the size of the portal region 433 is proportional to the size of the avatar 422 (and the size of the portal region is greater than the size of the avatar). For example, in various embodiments, the portal region 433 is 1.25 times as large, 1.5 times as large, 2 times as large, or 3 times as large as the avatar 422 (in any region or any particular dimension).

[0071] Figure 4G The CGR environment 400 at the third time is shown with a fourth focus conflict resolution. Figure 4AThe CGR environment 400. At the fourth time, an avatar 422 is displayed at a third position behind the wall 416. In response to determining that the distance to the avatar 422 is greater than the distance to the wall 416, the CGR environment 400 includes a virtual world 440 that covers the entire scene, thereby obscuring all real objects in the CGR environment 400. In various embodiments, the virtual world 440 includes representations of some real objects (e.g., the table 412 and the lamp 414), while excluding representations of other real objects (e.g., the wall 416 and the floor 417).

[0072] The virtual world 440 includes a virtual ground 428, virtual trees 424, a virtual sun 426, and an avatar 422 located at the third position. In various embodiments, the virtual world 440 includes a virtual shadow of the avatar 422 displayed on the virtual ground 428.

[0073] Figure 5 is a flowchart representation of a method 500 for resolving focus conflicts in a CGR environment according to some embodiments. In various embodiments, the method 500 is performed by a device having one or more processors, non-transitory memory, an image sensor, and a display (e.g., Figure 3 the electronic device 120 in). In some embodiments, the method 500 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 500 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0074] In block 510, the method 500 begins with the device using the image sensor to capture an image of a scene that includes a real object at a first distance in a particular direction. For example, in Figure 4A the wall 416 is at a certain distance from the scene camera in a particular direction.

[0075] In block 520, the method 500 continues with the device displaying a CGR environment on the display that includes virtual objects at a second distance from the device in a particular direction. For example, in Figure 4A the avatar 422 is displayed at a certain distance from the scene camera in a particular direction at the first position. As another example, in Figure 4D the avatar 422 is displayed at a certain distance from the scene camera in a particular direction at the third position.

[0076] In block 521, based on determining that the second distance is less than the first distance, the CGR environment includes virtual objects that cover the scene. For example, in Figure 4A based on determining that the distance to the avatar 422 is less than the distance to the wall 416, the CGR environment 400 includes the avatar 422 displayed over the scene (including the wall 416). As another example, inFigure 4B In Figure 4B , according to the determination that the distance to the avatar 422 is less than the distance to the wall 416, the CGR environment 400 includes an avatar 422 displayed above the scene (including the wall 416).

[0077] In box 522, according to the determination that the second distance is greater than the first distance, the CGR environment includes a virtual object with an obfuscation zone that obfuscates at least a portion of the real object within the obfuscation zone. In various embodiments, the obfuscation zone surrounds the virtual object. For example, in Figure 4D In

[0077] , according to the determination that the distance to the avatar 422 is greater than the distance to the wall 416, the CGR environment 400 includes an avatar 422 surrounded by a masking region 431 that hides at least a portion of the wall 416. For another example, in Figure 4E In Figure 4D , according to the determination that the distance to the avatar 422 is greater than the distance to the wall 416, the CGR environment 400 includes an avatar 422 surrounded by a blur region 432 that blurs at least a portion of the wall 416. For another example, in Figure 4F In Figure 4E , according to the determination that the distance to the avatar 422 is greater than the distance to the wall 416, the CGR environment 400 includes an avatar 422 surrounded by a portal region 433 that hides at least a portion of the wall 416. For another example, in Figure 4F In Figure 4F , according to the determination that the distance to the avatar 422 is greater than the distance to the wall 416, the CGR environment 400 includes an avatar 422 surrounded by a virtual world 440 that hides the wall 416 (and all other real objects in the scene).

[0078] In various embodiments, the obfuscation zone includes a blur region that blurs the portion of the real object within the blur region. For example, in Figure 4E In

[0078] , the avatar 422 is surrounded by a blur region 432 that blurs the portion of the wall 416 (and the TV 413) within the blur region 432. In various embodiments, the obfuscation zone includes a dimming region that dims the portion of the real object within the dimming region. In various embodiments, the amount of blurring and / or dimming is reduced further away from the virtual object.

[0079] In various embodiments, the obfuscation zone includes a masking region that occludes the portion of the real object within the masking region. For example, in Figure 4D In

[0079] , the avatar 422 is surrounded by a masking region 431 that masks, covers, and hides the portion of the wall 416 (and the TV 413) within the masking region 431.

[0080] In various embodiments, the obfuscation zone includes a portal region that displays a virtual world over the portion of the real object within the portal region. For example, in Figure 4FIn [description], the avatar 422 is surrounded by a portal region 433 that displays a virtual world. In various embodiments, the virtual world includes a virtual floor. For example, in Figure 4F In [description], the portal region 433 displays a virtual floor 435. In various embodiments, the virtual floor is coplanar with the real floor of the scene. For example, in Figure 4F In [description], the virtual floor 435 is coplanar with the floor 417. In various embodiments, the method 500 further includes displaying a virtual shadow of a virtual object on the virtual floor.

[0081] In various embodiments, the obfuscation region occupies the entire display. For example, in Figure 4G In [description], the avatar 422 is surrounded by a virtual world 440 that obscures the wall 416 and all other real objects of the scene. In various embodiments, the obfuscation region that occupies the entire display is a masking region, a blurring region, a dimming region, or a portal region.

[0082] In various embodiments, displaying a CGR environment includes displaying a representation of a scene on a display. Various focus conflict resolutions can be performed on a device with an opaque display. For example, the application of a blurring region can be performed on a device with an opaque display by displaying a blurred representation of the scene in the blurring region. Additionally, various focus conflict resolutions can be performed on a device with a transparent display. For example, the display of a masking region can be performed on a device with a transparent display by displaying a masking region around a virtual object.

[0083] Although the various aspects of the specific embodiments within the scope of the appended claims have been described above, it should be apparent that the various features of the above-described specific embodiments can be embodied in a wide variety of forms, and any such specific structure and / or function is merely illustrative. Based on this disclosure, those skilled in the art should understand that the aspects described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, a device can be implemented using any number of the aspects described herein and / or a method can be practiced. Additionally, other structures and / or functions can be used to implement such a device and / or to practice such a method in addition to or different from one or more of the aspects described herein.

[0084] It will also be understood that although terms such as "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. The first node and the second node are both nodes, but they are not the same node.

[0085] The terms used herein are merely for the purpose of describing particular embodiments and are not intended to limit the claims. As used in the description of the present embodiments and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0086] As used herein, the term "if" can be interpreted to mean "when the precondition is true" or "when the precondition is true" or "in response to determining" or "in accordance with determining" or "in response to detecting" the precondition is true, depending on the context. Similarly, the phrases "if it is determined that [the precondition is true]" or "if [the precondition is true]" or "when [the precondition is true]" are interpreted to mean "when it is determined that the precondition is true" or "in response to determining" or "in accordance with determining" the precondition is true or "when it is detected that the precondition is true" or "in response to detecting" the precondition is true, depending on the context.

Claims

1. A method, comprising: at a device including an image sensor, a display, one or more processors, and non-transitory memory: using the image sensor to capture an image of a first real environment including a real object at a first distance from the device in a specific direction; and displaying on the display a computer-generated reality (CGR) environment that includes a virtual avatar at a second distance from the device in the specific direction, wherein the virtual avatar represents a person in a second real environment from the first real environment, and the virtual avatar moves in the CGR environment corresponding to the movement of the person in the second real environment; wherein, based on determining that the second distance is less than the first distance, the CGR environment includes the virtual avatar overlaid on the first real environment; and wherein, based on determining that the second distance is greater than the first distance, the CGR environment includes the virtual avatar with an obfuscation zone that obfuscates at least a portion of the real object within the obfuscation zone.

2. The method according to claim 1, wherein the obfuscation zone surrounds the virtual avatar.

3. The method according to claim 1, wherein the obfuscation zone includes a blur region that blurs the portion of the real object within the blur region.

4. The method according to claim 3, wherein the virtual avatar is not blurred.

5. The method according to claim 1, wherein the obfuscation zone includes a dimming region that dims the portion of the real object within the dimming region.

6. The method according to claim 5, wherein the virtual avatar is not dimmed.

7. The method according to claim 1, wherein the obfuscation zone includes a masking region that masks the portion of the real object within the masking region.

8. The method according to claim 1, wherein the obfuscation zone includes a portal region that displays a virtual world over the portion of the real object within the portal region.

9. The method according to claim 8, wherein the virtual world includes a virtual floor.

10. The method according to claim 9, wherein the virtual floor is coplanar with the real floor of the scene.

11. The method according to claim 9, further comprising displaying a virtual shadow of the virtual avatar on the virtual floor.

12. A device, comprising: an image sensor; a display; non-transitory memory; and one or more processors configured to: use the image sensor to capture an image of a first real environment including a real object at a first distance from the device in a specific direction; and display on the display a computer-generated reality (CGR) environment that includes a virtual avatar at a second distance from the device in the specific direction, wherein the virtual avatar represents a person in a second real environment from the first real environment, and the virtual avatar moves in the CGR environment corresponding to the movement of the person in the second real environment; Wherein, based on determining that the second distance is less than the first distance, the CGR environment includes the virtual avatar overlaid on the first real environment; and Wherein, based on determining that the second distance is greater than the first distance, the CGR environment includes the virtual avatar with a confusion area that confuses at least a portion of the real object within the confusion area.

13. The apparatus according to claim 12, wherein the confusion area surrounds the virtual object.

14. The apparatus according to claim 12, wherein the confusion area includes a blur area that blurs the portion of the real object within the blur area.

15. The apparatus according to claim 12, wherein the confusion area includes a dimming area that dims the portion of the real object within the dimming area.

16. The apparatus according to claim 12, wherein the confusion area includes a masking area that masks the portion of the real object within the masking area.

17. The apparatus according to claim 12, wherein the confusion area includes a portal area that displays a virtual world over the portion of the real object within the portal area.

18. The apparatus according to claim 17, wherein the virtual world includes a virtual floor coplanar with the real floor of the scene.

19. The apparatus according to claim 18, wherein the one or more processors are further configured to: display a virtual shadow of the virtual avatar on the virtual floor.

20. A non-transitory memory storing one or more programs, the one or more programs, when executed by one or more processors of a device having an image sensor and a display, cause the device to: Capture, using the image sensor, an image of a first real environment including a real object at a first distance from the device in a specific direction; and Display, on the display, a computer-generated reality CGR environment including a virtual avatar at a second distance from the device in the specific direction, wherein the virtual avatar represents a person in a second real environment from the first real environment, and the virtual avatar moves correspondingly to the movement of the person in the second real environment in the CGR environment; Among them, Based on determining that the second distance is less than the first distance, the CGR environment includes the virtual avatar overlaid on the first real environment; and Wherein, based on determining that the second distance is greater than the first distance, the CGR environment includes the virtual avatar with a confusion area that confuses at least a portion of the real object within the confusion area.

Images