MANIPULATION OF VIRTUAL OBJECTS USING A TRACKED PHYSICAL OBJECT

Abstract

Method comprising: in an electronic device (120) with a display (1006): Presenting content (200) including a virtual object (130) and a virtual representation of a proxy device (150, 700, 800) physically associated with the electronic device (120) on the display (1006), wherein the proxy device (150, 700, 800) includes an input device on an outer surface of a housing (710), wherein the housing (710) defines a peripheral boundary of the proxy device (150, 700, 800), and wherein the peripheral boundaries of the proxy device (150, 700, 800) form a cube or a sphere and the virtual representation is a cube or a sphere, respectively; Receiving, from the proxy device (150, 700, 800), an input obtained using the input device, the input representing a request to create a fixed alignment between the virtual object (130) and the virtual representation (250) in a three-dimensional 3-D coordinate space defined for the content (200); generating the fixed orientation in response to receiving the input; and dynamically updating a position and an orientation of the virtual object in 3D coordinate space using position data representing the movement of the proxy device in a physical environment, wherein the position and orientation of the virtual object (130) are updated to maintain the fixed orientation.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 736,551, filed on Friday, January 25, 2019. TECHNICAL FIELD

[0002] The present disclosure relates generally to systems, methods, and apparatus for presenting content on a display of an electronic device, and more particularly to manipulating virtual objects using a tracked physical object during the presentation of content on a display of an electronic device. BACKGROUND

[0003] There are various electronic devices, such as data helmets (also known as headsets and HMDs), with displays that provide the user with a computer-generated reality (CGR) environment in which they can be fully immersed in a surrounding physical environment, fully immersed in an environment containing virtual objects, or somewhere in between. While direct manipulation of physical objects in the surrounding physical environment is of course possible for the user, this is not true for virtual objects in the CGR environment. The lack of a means to directly interact with virtual objects presented to a user as part of a CGR environment limits the extent to which the virtual objects are integrated into the CGR environment. Therefore, it may be desirable to provide users with a means to directly manipulate virtual objects presented as part of a CGR environment.

[0004] Van de Kerckhove, E.: Advanced VR Mechanics with Unity and the HTC Vive part 1; 2017, URL: https: / / www.raywenderlich.com / 625-advanced-vr-mechanics-with-unity-and-the-htcvive-part-1, 10.05.2021 concerns an advanced HTC Vive tutorial that explains an extensible interaction system and provides several ways to grasp and move virtual objects in this system.

[0005] US 2017 / 0 244 811 A1 relates to a system for pairing a first device and a second device in a virtual reality environment, wherein the first device may be a transmitting device and the second device may be a receiving device. The transmitting device may transmit an electromagnetic signal that is received by the receiving device. The receiving device may process the electromagnetic signal to verify the physical proximity of the receiving device and the transmitting device and to extract identification information related to the transmitting device for pairing. The receiving device may display one or more virtual pairing indicators that can be manipulated to verify the user's intent to pair the first and second devices.

[0006] Playstation Move In: Wikipedia, refers to the motion game controller developed by Sony. SUMMARY

[0007] The various implementations disclosed herein include devices, systems, and methods for manipulating virtual objects using a tracked physical object while simultaneously presenting content on a display of an electronic device. In one implementation, a method includes presenting content, including a virtual object and a virtual representation of a proxy device physically associated with the electronic device, on a display of an electronic device. The method comprises the features of claim 1.

[0008] In one example implementation, a non-transitory computer-readable storage medium stores program instructions executable by a processor to perform operations. The operations include presenting content, including a virtual object and a virtual representation of a proxy device that is not physically connected to an electronic device, on a display of an electronic device. The proxy device includes an input device disposed on an outer surface of a housing that defines the peripheral boundaries of the proxy device. The housing is adapted for manipulation by a gripping force exerted by a user's hand on diametrically opposed surface areas of the housing and on another surface area of ​​the housing positioned orthogonal to the diametrically opposed surface areas.Input is received from the proxy device via the input device, which represents a request to establish a fixed alignment between the virtual object and the virtual representation in a three-dimensional ("3D") coordinate space defined for the content. The fixed alignment is established in response to receipt of the input. The position and orientation of the virtual object in the 3D coordinate space are dynamically updated using position data that defines the movement of the proxy device in the physical environment.

[0009] In another example implementation, a proxy device includes a housing that defines the peripheral boundaries of the proxy device and an input device. The housing is symmetric with respect to each axis of a proxy device reference frame fixed at the center of the proxy device. The input device is disposed on an outer surface of the housing and configured to receive inputs representing requests for manipulation of a virtual object included in content presented on a display of an electronic device not physically connected to the proxy device, via a virtual representation of the proxy device in the content. Movement of the proxy device in a physical environment translates into movement of the virtual representation in a three-dimensional ("3-D") coordinate space defined for the content.

[0010] According to further example implementations, an electronic device includes one or more processors, memory, and one or more programs; the one or more programs are stored in non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the performance of each of the methods described herein. According to some implementations, instructions are stored in a non-transitory computer-readable storage medium that, when executed by one or more processors of a device, cause the device to perform or cause the performance of each of the methods described herein.According to some implementations, the apparatus includes: one or more processors, non-volatile memory, and means for performing or causing to be performed any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order that the present disclosure may be understood by those skilled in the art, a more detailed description may be provided with reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings. Fig. 1 shows a block diagram of an example operating environment according to some implementations. Fig. Figure 2 illustrates an example of displaying content that represents a view of a computer-generated reality environment on a display of an electronic device while the electronic device is located in a physical environment that is compatible with the Fig. 1 illustrated operating environment. Fig. 3 illustrates an example of updating the Fig. 2 illustrated content with visual feedback associated with a virtual representation of a tracked physical object to interact with virtual objects in the computer-generated reality environment. Fig. 4 illustrates an example of updating the Fig. 2 to create a fixed alignment between a virtual object and a virtual representation of a tracked physical object. Fig. 5 is a block diagram of the Fig. 1 illustrates an exemplary operating environment following the physical movement of a tracked physical object in a physical environment. Fig. 6 illustrates an example of dynamic updating of the content of Fig. 4 in response to the physical movement of the tracked physical object in the physical environment. Fig. Figure 7 illustrates an example of a tracked physical object in accordance with some implementations. Fig. Figure 8 illustrates another example of a tracked physical object in accordance with some implementations. Fig. 9 is a flowchart illustrating an example method for manipulating virtual objects using a tracked physical object while simultaneously displaying the content on a display of an electronic device. Fig. 10 is a block diagram of an example electronic device suitable for some implementations.

[0012] According to common practice, the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be expanded or reduced as desired for clarity. In addition, some of the drawings may not depict all components of a given system, method, or apparatus. Finally, like reference numerals may be used to refer to like features throughout the specification and figures. DESCRIPTION

[0013] Numerous details are described to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings illustrate merely some example aspects of the present disclosure and are therefore not to be considered limiting. Those skilled in the art will recognize that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits have not been described in exhaustive detail so as not to obscure more relevant aspects of the example implementations described herein.

[0014] With reference to Fig. 1, an example of an operating environment 100 for implementing aspects of the present disclosure is illustrated and generally referred to as 100. In general, the operating environment 100 illustrates an electronic device 120 configured to present content to a user corresponding to a computer-generated reality environment. As used herein, a physical environment refers to a world that an individual can perceive and / or interact with without assistance from electronic systems. Physical environments (e.g., a physical forest) include physical elements (e.g., physical trees, physical structures, and physical animals). Individuals can directly interact with and / or perceive the physical environment, e.g., through touch, sight, smell, hearing, and taste.

[0015] A computer-generated reality (CGR) environment refers to a fully or partially simulated environment that humans perceive and / or interact with via an electronic system. In CGR, a subset of a person's physical movements or representations thereof are tracked, and in response, one or more properties of one or more virtual objects simulated in the CGR environment are adjusted to behave according to at least one physical law. For example, a CGR system can detect a person's head rotation and, in response, adjust the graphical content presented to the person and an auditory field in a similar way to how such views and sounds would change in a physical environment. In some situations (e.g.,For accessibility reasons, adjustments can be made to the features of virtual objects in a CGR environment in response to representations of physical movements (e.g., voice commands).

[0016] A person may perceive and / or interact with a CGR object using any of their senses, including sight, hearing, touch, taste, and smell. For example, a person may perceive and / or interact with audio objects that create a 3D or spatial audio environment, allowing the perception of point audio sources in 3D space. In another example, audio objects may enable audio transparency, selectively incorporating ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may only perceive and / or interact with audio objects.

[0017] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes virtual objects that a person can interact with and / or perceive. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can perceive and / or interact with virtual objects in the VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0018] In contrast to a VR environment, which relies entirely on computer-generated sensory inputs, a mixed reality (MR) environment refers to a simulated environment designed to incorporate sensory inputs from the physical environment or a representation of it in addition to computer-generated sensory inputs (e.g., virtual objects). On a virtuality continuum, a mixed reality environment exists somewhere between, but is not limited to, a fully physical environment at one end and a virtual reality environment at the other.

[0019] In some MR environments, computer-generated sensory inputs can respond to changes in sensory input from the physical environment. Furthermore, some electronic systems for representing an MR environment can track location and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items from the physical environment or their representations). For example, a system can account for motion so that a virtual tree appears stationary relative to the physical Earth.

[0020] Examples of mixed realities include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may include a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display so that a person using the system perceives the virtual objects overlaid on the physical environment. Alternatively, a system may include an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment.The system composites the images or videos with virtual objects and displays the composite on the opaque display. A person using the system indirectly views the physical environment via the images or video of the physical environment and perceives the virtual objects superimposed over the physical environment. As used herein, video of the physical environment displayed on an opaque display is called "pass-through video," which means that a system uses one or more image sensors to capture images of the physical environment and uses those images to display the AR environment on the opaque display. Alternatively, a system may include a projection system that projects virtual objects into the physical environment, e.g., as a hologram or onto a physical surface, so that a person using the system perceives the virtual objects superimposed over the physical environment.

[0021] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in the delivery of pass-through video, a system may transform one or more sensor images to establish a selected perspective (e.g., a viewpoint) that differs from the perspective captured by the image sensors. As another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions of it so that the modified portion may be representative but not photorealistic versions of the originally captured images. As another example, a representation of a physical environment may be transformed by graphically eliminating or obscuring portions of it.

[0022] An augmented virtuality (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but people with faces are photorealistically reproduced from images of physical people. As another example, a virtual object can take on a shape or color of a physical object imaged by one or more imaging sensors. As another example, a virtual object can take on shadows that match the position of the sun in the physical environment.

[0023] There are many different types of electronic systems that enable a person to perceive and / or interact with various CGR environments. Examples include data helmet systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display functionality, windows with integrated display functionality, displays designed as lenses intended to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earbuds, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A data helmet system may include one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accommodate an external opaque display (e.g., a smartphone).The head-mounted system may include one or more imaging sensors to capture images or video of the physical environment and / or one or more microphones to capture audio of the physical environment. Unlike an opaque display, a data helmet system may have a transparent or translucent display. The transparent or translucent display may include a medium through which light representative of images is directed toward a person's eyes. The display may use digital light projection, OLEDs, LEDs, uLEDs, liquid crystals on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical fiber, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to selectively become opaque.Projection-based systems can use retinal projection technology to project graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface.

[0024] Proxy device 150 is an example of a type of input mechanism that can be used to manipulate or otherwise interact with virtual objects (e.g., virtual object 130) as part of a CGR environment. As described in Fig. 1, proxy device 150 is an example of an input mechanism that is not physically connected to electronic device 120. In accordance with implementations of the present disclosure, one aspect of manipulating virtual objects with proxy device 150 is to create a fixed alignment between a virtual object and a virtual representation of proxy device 150. In creating such fixed alignments, proxy device 150 serves as a proxy in a physical environment for virtual objects in a CGR environment by directly translating the physical movement of proxy device 150 into computer-generated movement of the virtual objects. In one implementation, proxy device 150 includes a haptic device configured to actuate when proxy device 150 receives feedback signals associated with manipulations to virtual objects (e.g.,virtual object 130).

[0025] Fig. Figure 2 illustrates an example of the presentation of content 200 comprising a CGR environment on a display (e.g., the one or more displays 1006 in Fig. 10) of the electronic device 120. The content 200 generally corresponds to a view of the CGR environment that a user of the electronic device 120 may perceive before manipulating the virtual object 130 via the proxy device 150. As in Fig. 2, the content 200 includes the content corresponding to the virtual object 130 and the content representing the physical object 140. While the user can change a pose (e.g., a position and / or an orientation) of the physical object 140 in the physical environment through direct or indirect physical interaction, this does not apply to the virtual object 130. To change a pose of the virtual object 130, a proxy in the CGR environment is involved in transferring the physical interaction from a physical environment to a simulated interaction in the CGR environment represented by the content 200. For this purpose, the content 200 further includes a computer-generated sensory representation of the proxy device 150, which in Fig. 2 as a virtual representation 250 to serve as this proxy in the CGR environment.

[0026] In general, the physical movement of the proxy device 150 in the physical environment (e.g., scene 105) is directly translated into a simulated (or virtual) movement of the virtual representation 250 in the CGR environment. In other words, a physical movement of the proxy device 150 that changes a pose (e.g., a position and / or an orientation) of the proxy device 150 in the physical environment also changes a pose of the virtual representation 250 in a three-dimensional ("3-D") coordinate space of the CGR environment. The simulated movement, which translates the physical movement of the proxy device 150 to the virtual representation 250, can also be translated into a simulated movement of other virtual objects in the CGR environment by creating a fixed orientation between the virtual representation 250 and a virtual object.

[0027] For example, a CGR environment module (e.g. CGR environment module 1040 in Fig. 10) of the electronic device 120 may display the content 200 on a display of the electronic device 120. During the display of the content 200, an input representing a request to establish a fixed alignment between the virtual object 130 and the virtual representation 250 may be received from a user of the electronic device 120. In one implementation, the input is received by the proxy device 150 when at least a subset of the virtual representation 250 overlaps a location on the display occupied by the virtual object 130, as in Fig. 2. In one implementation, input is received by proxy device 150 when there is no overlap between virtual representation 250 and virtual object 130 on the display.

[0028] To receive such inputs from the user, the proxy device 150 includes an input device disposed on an outwardly facing surface of a housing that defines the peripheral boundaries of the proxy device 150. Examples of such input devices and housings are described below with reference to Fig. 7 and Fig. 8. After receiving the user input, the proxy device 150 sends the input via a data acquisition unit (e.g., data acquisition unit 1042 in Fig. 10) of the electronic device 120 to the CGR environment module for further processing. In response to receiving the input, a CGR presentation unit (e.g., CGR presentation unit 1044 in Fig. 10) of the CGR environment module, the fixed alignment between the virtual object 130 and the virtual representation 250 on the display.

[0029] Creating a fixed alignment between a virtual object and a virtual representation of a proxy device typically involves transitioning the virtual object from a pre-alignment pose to a post-alignment pose. A comparison between Fig. 2 and Fig. Figure 4 illustrates an example of this transition. In this example, the respective positions and orientations of the virtual object 130 and the virtual representation 250 in the content 200 are different when the virtual object 130 is in a Fig. 2. To achieve the Fig. 4, the CGR presentation unit determines display coordinates that define an anchor point 255 of the virtual representation 250 in a 3D coordinate space of the CGR environment. In one implementation, the display coordinates that define the anchor point 255 of the virtual representation 250 are based on a proxy reference frame 155 that is fixed in the physical environment at the center of the proxy device 150. The CGR presentation unit then overwrites the display coordinates that define an anchor point 235 of the virtual object 130 in the 3D coordinate space with the display coordinates corresponding to the anchor point 255 to generate the content 400. As a result, the respective positions and orientations of the virtual object 130 and the virtual representation 250 coincide after the virtual object 130 is moved into the Fig. 4 illustrated re-alignment pose.

[0030] Examples include the Fig. 2 and Fig. 4, the virtual object 130 and the virtual representation 250 are shown as a rectangular prism and a cube. If the virtual object 130 in this example is Fig. 4, the edges of the rectangular prism (representing the virtual object 130) and the cube (representing the virtual representation 250) are parallel. This example illustrates that, in one implementation, a body of the proxy object 150 is symmetrical with respect to one or more axes of a device reference frame (e.g., device reference frame 730 in Fig. 7) that is fixed at the center of the proxy device 150. In response to receiving a request to create the fixed alignment, the CGR presentation engine may snap an axis of the virtual object 130 to a direction vector of the virtual representation 250 that corresponds to an axis of the proxy device 150 based on proximity. That is, in this example, the CGR presentation engine creates the fixed alignment between the virtual object 130 and the virtual representation 250 by aligning the respective anchor points (anchor point 235 and anchor point 255) and snapping a closest match between each axis of the proxy device 150 (based on a direction vector of the virtual representation 250) and each axis of the virtual object 130 within the CGR environment.The fixed orientation between the virtual object 130 and the virtual representation 250 is generated without the proxy device 150 having a predefined forward / left / right / backward direction.

[0031] While the CGR presentation unit uses the respective anchor points of the virtual object 130 and the virtual representation 250 to create the fixed alignment in this example, one of ordinary skill in the art will recognize that other implementations are not so limited. In some implementations, the attachment properties of the virtual object 130 specify other ways of creating the fixed alignment between the virtual object 130 and the virtual representation 250. For example, the attachment properties of the virtual object 130 may specify that creating the fixed alignment includes snapping a particular surface (e.g., a floor surface) of the virtual object 130 to a particular location of the virtual representation 250. In this example, when the proxy device 150 communicates with the proxy device 700 of Fig. 7, the respective location of the virtual representation 250 may correspond to a horizontal surface (e.g., surface 710C) of the proxy device in the physical environment (e.g., scene 105). In one implementation, the source data for rendering the virtual object defines the mounting properties of the virtual object. In one implementation, rules encapsulated in computer-executable program instructions corresponding to the CGR presentation unit define the mounting properties of the virtual object.

[0032] In some implementations, visual feedback associated with a virtual representation can be used to mitigate a user's perception of unnatural movements as a virtual object transitions from a pre-alignment pose to a post-alignment pose. An example of such visual feedback is described in Fig. 3 by the visual feedback elements 355. Content 300 represents a view of the CGR environment presented on the display of the electronic device 120 between content 200 and content 400. A comparison between the Fig. 2-4 shows that the visual feedback elements 355 associated with the virtual representation 250 interact with the virtual object 130 to gradually move the virtual object 130 from the pre-alignment pose of the Fig. 2 into the re-alignment pose of the Fig. 4 to be transferred.

[0033] As explained above, the simulated motion that translates the physical motion of the proxy device 150 to the virtual representation 250 can also be translated into a simulated motion of other virtual objects in the CGR environment by creating a fixed alignment between the virtual representation 250 and a virtual object. To illustrate such a translation of the simulated motion, Fig. 5 illustrates the proxy device 150 while the content 400 is presented on the display of the electronic device 120. A comparison between Fig. 1 and Fig. 5 shows that the user of the electronic device 120 has changed a pose of the proxy device 150 in the physical environment. As in Fig. 6, the CGR presentation unit of the electronic device 120 has dynamically updated a pose of the virtual representation 250 in the CGR environment to reflect this pose change of the proxy device 150 in the physical environment. Fig. 6 further illustrates that the CGR presentation unit of the electronic device 120 has dynamically updated a pose of the virtual object 130 in the CGR environment to reflect this pose change of the proxy device 150 in the physical environment due to its fixed alignment with the virtual representation 250.

[0034] In various implementations, the CGR presentation unit dynamically updates a pose of the virtual object 130 or virtual representation 250 in the CGR environment to reflect any pose change of the proxy device 150 using position data defining the movement of the proxy device 150 in the physical environment. The position data may define the translational movement of the proxy device 150 in the physical environment, the rotational movement of the proxy device 150 in the physical environment, or a combination thereof. This position data may be obtained using any position tracking technique known to one of ordinary skill in the art. For example, the position data may be obtained using inertial data, image data, or a combination thereof.

[0035] In one implementation, the inertial data is obtained via an inertial measurement unit ("IMU") of the proxy device 150. Generally, an IMU is a computing device that provides inertial data by sampling signals generated by one or more sensors of the IMU. The one or more sensors may include: one- to three-axis accelerometers for measuring velocity changes along one to three axes, one- to three-axis gyroscopes for measuring angular velocity about one to three axes, one- to three-axis magnetometers for measuring information about magnetic fields related to one to three axes, a barometric pressure sensor for measuring air pressure, and the like.

[0036] In one implementation, the image data represents a plurality of optical sources configured to emit light. In one implementation, the plurality of optical sources are arranged on an outward-facing surface of the proxy device 150 at known locations relative to a proxy device reference frame (e.g., proxy device reference frames 155 and 730 of Fig. 1 and 7, respectively). In one implementation, the plurality of optical sources are arranged on an outwardly facing surface of the electronic device 120 at known locations relative to a device reference frame (e.g., device reference frame 125 of Fig. 1) arranged.

[0037] Fig. 7 and Fig. 8 illustrate two exemplary configurations of a proxy device suitable for implementing the proxy device 150 according to implementations of the present disclosure. A proxy device according to implementations of the present disclosure includes a housing defining the peripheral boundaries of the proxy device and one or more input devices disposed on an outward-facing surface of the proxy device. Each input device among the one or more input devices is configured to receive inputs representing requests corresponding to manipulations of a virtual object within a CGR environment.For example, an input device may receive input representing a request to establish a fixed alignment between the virtual object and a virtual representation of the proxy device and the virtual objects within a CGR environment.

[0038] A form of peripheral boundaries defined by the housing can facilitate establishing fixed alignments between a virtual representation of the proxy device and virtual objects within a CGR environment at specific steps. For example, the peripheral boundaries defined by the housing 710 in Fig. 7, a cube. By defining the peripheral boundaries of the proxy device 700 as a cube, the housing 710 facilitates the creation of fixed alignments between a virtual representation of the proxy device 700 and virtual objects within a CGR environment in 90-degree increments.

[0039] In one implementation, the housing is symmetrical with respect to one or more axes of a proxy device reference frame fixed at a center of the proxy device. In one implementation, the one or more user input devices include one or more physical input devices 720A-C / 820A-C. The input devices may include a physical input device (e.g., a button, a joystick, a switch, a knob, a dial, a touchpad, and the like). In one implementation, the one or more user input devices may include a simulated input device (e.g., a softkey, a virtual keyboard, and the like).In one implementation, the one or more user input devices are buttons configured to receive inputs representing a same request to map the movement of a virtual object to the movement of a virtual representation of the proxy device.

[0040] In this example, content including a virtual object having a bottom surface and an "upward" direction vector defined normal to the bottom surface may be presented on a display to a user interacting with proxy device 700. While the content is being presented to the user, proxy device 700 may receive a first input corresponding to an interaction with input device 720B. In response to receiving the first input, a CGR presentation unit (e.g., CGR presentation unit 1044 in Fig. 10) update the content to create a fixed orientation in which the "up" direction vector is parallel to a virtual representation direction vector corresponding to a y-axis of the proxy device reference frame 730.

[0041] Alternatively, or subsequent to receiving the first input, the proxy device 700 may receive a second input corresponding to an interaction with the input device 720C while presenting the content to the user. In response to receiving the second input, the CGR presentation unit may update the content to create a fixed orientation in which the "up" direction vector is parallel to a direction vector of the virtual representation corresponding to a z-axis of the proxy device 730, which is fixed at the center 735 of the proxy device 700.

[0042] In another example, the proxy device 700 may receive input corresponding to one of the input devices 720A-C while presenting the content to the user. In response to receiving the input, the CGR presentation engine may determine a closest match between each axis of a virtual representation of the proxy device 700 and each axis of a virtual object within the CGR environment. For example, an x-axis of the virtual object may be closer to a direction vector corresponding to a z-axis of the proxy device reference frame 730 than direction vectors corresponding to an x-axis or a y-axis of the proxy device reference frame 730. In this case, the CGR presentation engine may update the content to create a fixed alignment between the x-axis of the virtual object and the direction vector corresponding to the z-axis of the proxy device reference frame 730.In one implementation, the CGR presentation engine may generate this fixed orientation regardless of which input device was actuated to generate the input. In this implementation, the CGR presentation engine would generate the fixed orientation between the x-axis of the virtual object and the direction vector corresponding to the z-axis of the proxy device reference frame 730, even if the user actuated input device 720A or input device 720B, which intersect the x-axis or y-axis, respectively, of the proxy device reference frame 730 to generate the input.

[0043] While the peripheral boundaries defined by the housing 710 in Fig. 7 form a cube, one skilled in the art will recognize that other implementations are not so limited. According to some implementations, the peripheral boundaries of a proxy device defined by its housing may take other shapes. For example, Fig. 8 illustrates the peripheral boundaries defined by the housing 810 as forming a sphere. As another example, the peripheral boundaries of a proxy device defined by its housing may form a truncated cube.

[0044] In one implementation, each axis of a proxy device reference frame intersects with at least one of the one or more input devices disposed on an outward-facing surface of a proxy device. In Fig. 7, for example, the x-axis, y-axis, and z-axis of the proxy device frame 730 intersect with the input device 720A, the input device 720B, and the input device 720C. In one implementation, each axis of a proxy device reference frame intersects a plurality of input devices. For example, in Fig. 8 shows a first axis 830 of a proxy device reference frame corresponding to the proxy device 800, with the input device 820A and the input device 820B on diametrically opposite surface areas of the housing 810.

[0045] Manipulation of the proxy device by the user may be facilitated by the physical dimensions of the peripheral boundaries defined by the housing. To this end, the housing is adapted for manipulation by a gripping force exerted by the user's hand on diametrically opposed surface areas of the housing and on another surface area of ​​the housing positioned orthogonally to the diametrically opposed surface areas. Fig. 8, for example, the housing 810 is adapted for manipulation by a gripping force exerted by a user's hand on diametrically opposed surface areas of the housing 810 proximate the input devices 820A-820B and on another surface area (e.g., a surface area proximate the input device 820C) of the housing 810 positioned orthogonally to these diametrically opposed surface areas.

[0046] In one implementation, the one or more input devices include an input device configured to receive at least one input defined by a characteristic of an interaction with the input device. For example, if the input device is a touchpad, a characteristic of an interaction with the input device may include a duration of time an object remained in contact with the touchpad, an orientation of the object when touching the touchpad, a region of the touchpad where contact between the object and the touchpad occurred, any motion or acceleration information associated with the object when touching the touchpad, and the like.

[0047] Fig. 9 is a flowchart illustrating an example of a method 900 for manipulating virtual objects using a tracked physical object while simultaneously presenting the content on a display of an electronic device. At block 902, the method 900 includes presenting content, including a virtual object and a virtual representation of a proxy device that is not physically connected to the electronic device, on the display. The proxy device includes an input device disposed on an outer surface of a housing that defines the peripheral boundaries of the proxy device. In one implementation, the housing is symmetric with respect to one or more axes of a proxy device reference frame fixed at the center of the proxy device.In one implementation, the housing is designed for manipulation by a gripping force exerted by a user's hand on diametrically opposed surface areas of the housing and on another surface area of ​​the housing positioned orthogonally to the diametrically opposed surface areas.

[0048] At block 904, method 900 includes receiving input from the proxy device via an input device representing a request to create a fixed alignment between the virtual object and the virtual representation in a 3D coordinate space defined for the content. In one implementation, the input representing the request to create the fixed alignment is received by the proxy device when at least a subset of the virtual representation overlaps a location on the display occupied by the virtual object.

[0049] At block 906, method 900 includes generating the fixed orientation in response to receiving the input from the proxy device. In one implementation, generating the fixed orientation includes snapping a first surface of the virtual object to a portion of the virtual representation that corresponds to a horizontal surface of the proxy device in the physical environment. In one implementation, the source data for rendering the virtual object defines the first surface as a bottom surface of the virtual object. In one implementation, the source data for rendering the virtual object defines attachment properties of the virtual object.

[0050] In one implementation, generating the fixed orientation includes updating the content presented on the display to include a visual feedback element associated with the virtual representation. In one implementation, the visual feedback element interacts with the virtual object on the display to progressively transition the virtual object from a pre-alignment pose to a post-alignment pose according to the fixed orientation. In one implementation, the fixed orientation between the virtual object and the virtual representation is maintained for the duration of an interaction with the input device generating the input. In one implementation, the fixed orientation between the virtual object and the virtual representation is maintained after completion of an interaction with the input device generating the input.

[0051] In block 908, the method 900 includes dynamically updating a pose (e.g., a position, an orientation, or a combination thereof) of the virtual object in 3D coordinate space using position data defining the movement of the proxy device in a physical environment. In one implementation, the position data defines the translational and rotational movement of the proxy device in the physical environment.

[0052] In one implementation, method 900 further includes ending the fixed alignment to release the virtual object from the virtual representation of the proxy device. In one implementation, the fixed alignment is ended in response to receiving another input representing a request to end the fixed alignment via an input device of the proxy device after creating the fixed alignment. In one implementation, the input representing the request to create the fixed alignment and the subsequent input representing the request to end the fixed alignment are received via the same input device of the proxy device.In one implementation, the input representing the request to establish the fixed alignment and the subsequent input representing the request to terminate the fixed alignment are received using different input devices of the proxy device. In one implementation, the fixed alignment is terminated in response to a release of the input device used to receive the input representing the request to create the fixed alignment.

[0053] In one implementation, the position data is obtained using an image sensor, an optical source, an electromagnetic source, an electromagnetic sensor, an inertial measurement unit ("IMU"), or a combination thereof. In one implementation, at least one of the image sensor, optical source, electromagnetic source, and electromagnetic sensor is a component of the proxy device. In one implementation, at least one of the image sensor, optical source, electromagnetic source, and electromagnetic sensor is a component of the electronic device.

[0054] In one implementation, method 900 is performed by processing logic, including hardware, firmware, software, or a combination thereof. In one implementation, method 900 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory).

[0055] Fig. 10 is a block diagram of an electronic device 120 suitable for implementing aspects of the present invention. While certain features are illustrated, those skilled in the art will appreciate from this disclosure that various other features have not been illustrated for brevity so as not to obscure more pertinent aspects of the implementations disclosed herein. To this end, in some implementations, the electronic device 120 includes, by way of non-limiting example, one or more processors 1002 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, compute cores, or the like), one or more I / O devices 1004, one or more displays 1006, one or more communication interfaces 1008 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or similar interface types), one or more programming interfaces (e.g., I / O) 1010, memory 1020, and one or more communication buses 1050 for interconnecting these and various other components. In short, a GPU may include specialized electronic circuitry configured to quickly manipulate and modify memory to accelerate the generation of images in a frame buffer. A GPU may be present on a video card, or it may be embedded in a motherboard or, in certain CPUs, in the CPU die.

[0056] The one or more I / O devices 1004 are configured to form a human-machine interface that exchanges commands, requests, information, data, and the like between the electronic device 120 and a user. The one or more I / O devices 1004 may include, but are not limited to, a keyboard, a pointing device, a microphone, a joystick, and the like. In some implementations, the one or more I / O devices 1004 include at least one of an inertial measurement unit (“IMU”), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic motor, one or more depth sensors (e.g.,a structured light, a flight time or the like) and / or the like.

[0057] In one implementation, the one or more I / O devices 1004 include an image sensor configured to receive image data representing a physical environment in the electronic device 120. Examples of suitable image sensors for implementing image sensors of the one or more I / O devices 1004 may include one or more RGB cameras (e.g., with a complementary metal-oxide-semiconductor ("CMOS") image sensor or a charge-coupled device (CCD) image sensor), a monochrome camera, an IR camera, an event-triggered camera, or the like.

[0058] In some implementations, the one or more displays 1006 are configured to present the content to the user as described above. In some implementations, the one or more displays 1006 correspond to a holographic display, DLP (digital light processing) display, LCD (liquid crystal display), LCoS (liquid crystal on silicon) display, OLET (organic light-emitting field-effect transient display), OLED (organic light-emitting diode) display, SED (surface-conduction electron-emitter) display, FED (field-emission display), QD-LED (quantum-dot light-emitting diode), MEMS (micro-electro-mechanical system) display, or similar display types. In some implementations, the one or more displays 1006 correspond to diffractive, reflective, polarized, holographic, etc., waveguide displays.

[0059] In one implementation, the one or more displays 1006 include a display device comprising a plurality of pixels. Each pixel among the plurality of pixels can be implemented using LED (light-emitting diode), organic OLED (organic light emitting diode), plasma cell, LCD (liquid crystal display) components, and the like. In one implementation, one or more displays 1006 are disposed on an inward-facing surface of the electronic device 120. In one implementation, the one or more displays 1006 are a see-through display through which a portion of the physical environment is visible.

[0060] In one implementation, the one or more displays 1006 include a stereoscopic image display for presenting left and right eye views. In one implementation, the stereoscopic image display presents a stereoscopic subset of a 3-D representation of a scene corresponding to a physical environment (e.g., scene 105 in Fig. 1) in which the electronic device 120 is located. In one implementation, the 3D representation of the scene is reconstructed from light field images captured by an array of image sensors included in one or more I / O devices 1004.

[0061] The one or more communication interfaces 1008 may include any device or group of devices capable of establishing a wired or wireless data or telephone connection to one or more networks. Non-limiting examples of a network interface device include an Ethernet network adapter, a modem, or the like. A device may transmit messages as electronic or optical signals.

[0062] The one or more programming interfaces (e.g., I / O interfaces) 1010 are configured to communicatively couple the one or more I / O devices 1004 to other components of the electronic device 120. Thus, the one or more programming interfaces 1010 are capable of accepting commands or inputs from a user via the one or more I / O devices 1004 and transmitting the inputs to the one or more processors 1002.

[0063] Memory 1020 may include any suitable computer-readable medium. A computer-readable storage medium should not be understood as transient signals per se (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating via a transmission medium such as a waveguide, or electrical signals transmitted via a cable). For example, memory 1020 may include high-speed random access memory, such as DRAM, SRAM, DDR-RAM, or other solid-state random access memory devices. In some implementations, memory 1020 optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory, or other non-volatile solid-state memory devices.Memory 1020 optionally includes one or more storage devices located remotely from the one or more processing units 1002. Memory 1020 comprises a non-transitory computer-readable storage medium. Instructions stored in memory 1020 may be executed by the one or more processors 1002 to perform a variety of methods and operations, including the techniques for manipulating virtual objects using a tracked physical object while presenting the content on a display of an electronic device, which were described in more detail above.

[0064] In some implementations, the memory 1020 or the non-transitory computer-readable storage medium of the memory 1020 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 1030 and a CGR environment module 1040. The operating system 1030 includes processes for handling various basic system services and for performing hardware-dependent tasks. In some implementations, the CGR environment module 1040 is configured to manage and coordinate one or more CGR environments for one or more users (e.g., a single CGR environment for one or more users or multiple CGR environments for respective groups of one or more users).For this purpose, the CGR environment module 1040 includes, in various implementations, a data acquisition unit 1042, a CGR presentation unit 1044, a proxy device tracking unit 1046, and a data transmission unit 1048.

[0065] In some implementations, the data acquisition unit 1042 is configured to receive data (e.g., location data, inputs from proxy devices, content from a graphics pipeline, etc.) from one or more computing devices external to the electronic device 120. To this end, the data acquisition unit 1042 includes, in various implementations, instructions or logic therefor, as well as heuristics and metadata therefor.

[0066] In some implementations, the CGR presentation unit 1044 is configured to present CGR content via the one or more displays 1006. To this end, the CGR presentation unit 1044 includes, in various implementations, instructions or logic therefor, as well as heuristics and metadata therefor.

[0067] In some implementations, the proxy device tracking unit 1046 is configured to determine a pose (e.g., a position, an orientation, or a combination thereof) of a proxy device in a physical environment proximate the electronic device 120 based on the position data received from the data acquisition unit 1042. To this end, the proxy device tracking unit 1046 includes, in various implementations, instructions or logic therefor, as well as heuristics and metadata therefor.

[0068] In some implementations, the data transfer unit 1048 is configured to transfer data (e.g., location data, inputs from proxy devices, requests for content updates from a graphics pipeline, etc.) to one or more computing devices external to the electronic device 120. For these purposes, the data transfer unit 1048 includes, in various implementations, instructions or logic therefor, as well as heuristics and metadata therefor.

[0069] Although the data acquisition unit 1042, the CGR presentation unit 1044, the proxy device tracking unit 1046, and the data transmission unit 1048 are shown as being located on a single device (e.g., the electronic device 120), it should be understood that in other implementations, any combination of the data acquisition unit 1042, the CGR presentation unit 1044, the proxy device tracking unit 1046, and the data transmission unit 1048 may be located on separate computing devices.

[0070] Fig. Figure 10 is intended as a functional description of the various features present in a particular implementation, rather than a structural scheme of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately could be combined and some elements separated. For example, some functional modules shown separately in Fig.10 may be implemented in a single module, and the various functions of individual functional blocks could be implemented by one or more functional blocks in different implementations. The actual number of modules and the distribution of specific functions and how features are assigned to them vary from one implementation to another and, in some implementations, depend in part on the particular combination of hardware, software, or firmware chosen for a particular implementation.

[0071] The use of "adapted for" or "configured to" herein is intended as open and inclusive language that does not exclude devices adapted to, or configured to perform, additional tasks or steps. Additionally, the use of "based on" is intended to be open and inclusive in that a process, step, calculation, or other action based on the one or more stated conditions or values ​​may, in practice, be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for convenience of explanation only and are not intended to be limiting.

[0072] It should also be understood that while the terms "first," "second," etc., may be used herein to describe various elements, those elements are not restricted by these terms. These terms are used only to distinguish one element from another. For example, a first node could be referred to as a second node, and similarly, a second node could be referred to as a first node without changing the meaning of the description, as long as each occurrence of the "first node" is consistently renamed and each occurrence of the "second node" is consistently renamed. The first node and the second node are both nodes, but they are not the same node.

[0073] The terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the claims. As used in the description of the implementations and the appended claims, the singular forms "a", "an", "the", "the", and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It is also to be understood that the term "or," as used herein, refers to and includes any and all combinations of one or more of the related recited terms. It is further to be understood that the terms "comprises" and "comprising," when used in this specification, imply the presence of recited features, integers, steps, acts, elements, andSpecify components, but do not exclude the presence or addition of one or more other characteristics, integers, steps, operations, elements, components, or groups thereof.

[0074] As used herein, the term "if" may be understood as "when" or "when" or "as a result of determining" or "according to a determination" or "as a result of recognizing" that a stated antecedent condition is met, depending on the context. Similarly, the phrase "when it is determined [that a stated antecedent condition is met]" or "if [a stated antecedent condition is met]" or "when [a stated antecedent condition is met]" may be interpreted as "upon determining" or "as a result of determining that" or "according to a determination" or "upon recognizing" or "as a result of recognizing" that a stated antecedent condition is met, depending on the context.

[0075] The foregoing description and summary of the invention are to be considered in all respects as illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein should be determined not only from the detailed description of the illustrative implementations, but according to the full breadth permitted by patent law. It is to be understood that the implementations shown and described herein are merely illustrative of the principles of the present invention, and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

[1] Method comprising: in an electronic device (120) with a display (1006): Presenting content (200) including a virtual object (130) and a virtual representation of a proxy device (150, 700, 800) physically associated with the electronic device (120) on the display (1006), wherein the proxy device (150, 700, 800) includes an input device on an outer surface of a housing (710), wherein the housing (710) defines a peripheral boundary of the proxy device (150, 700, 800), and wherein the peripheral boundaries of the proxy device (150, 700, 800) form a cube or a sphere and the virtual representation is a cube or a sphere, respectively; Receiving, from the proxy device (150, 700, 800), an input obtained using the input device, the input representing a request to create a fixed alignment between the virtual object (130) and the virtual representation (250) in a three-dimensional 3-D coordinate space defined for the content (200); generating the fixed orientation in response to receiving the input; and dynamically updating a position and an orientation of the virtual object in 3D coordinate space using position data representing the movement of the proxy device in a physical environment, wherein the position and orientation of the virtual object (130) are updated to maintain the fixed orientation. [2] The method of claim 1, wherein the housing (710) is symmetrical with respect to each axis of a proxy device reference frame fixed at the center of the proxy device (150, 700, 800). [3] The method of claim 1, wherein creating the fixed alignment includes: Snapping a first surface of the virtual object (130) to a portion of the virtual representation (250) that corresponds to a horizontal surface of the proxy device (150, 700, 800) in the physical environment. [4] The method of claim 3, wherein source data for rendering the virtual object (130) defines the first surface as a bottom surface of the virtual object (130). [5] The method of claim 1, wherein creating the fixed alignment includes: Snapping a second surface of the virtual object (130) to a portion of the virtual representation (250) that corresponds to a vertical surface of the proxy device (150, 700, 800) in the physical environment, or snapping an axis of the virtual object (130) to a direction vector of the virtual representation (250) based on proximity. [6] The method of claim 1, wherein creating the fixed alignment includes: Updating the content presented on the display (1006) to include a visual feedback element associated with the virtual representation (250) that interacts with the virtual object (130) on the display (1006) to gradually transition the virtual object (130) from a pre-alignment pose to a post-alignment pose according to the fixed orientation. [7] The method of claim 1, wherein the housing (710) is configured to be grasped by a user's hand at diametrically opposed surface areas of the housing (710) while contacting a palm of the hand at another surface area of ​​the housing (710) positioned orthogonal to the diametrically opposed surface areas.

Images