Method and apparatus for virtual object manipulation in a physical environment

Abstract

A method, computing device, and head-mounted display device for manipulating virtual objects displayed via a display device are disclosed. In one example, image data of a physical environment including physical features is received. A three-dimensional model of at least a portion of the environment is generated. Candidate anchor features are extracted from the image data, each candidate anchor feature corresponding to one of the physical features. User input to manipulate a virtual object displayed within the environment is received. Based on the manipulation, a correspondence between a virtual anchor feature of the virtual object and a corresponding candidate anchor feature is identified. An indication of the corresponding candidate anchor feature is displayed to the user at the corresponding physical feature within the environment.

Description

Background Technology

[0001] Augmented reality display devices enable users to view, create, and manipulate virtual content displayed to appear within their physical environment. In some examples, users may want to locate and orient virtual objects or portions thereof relative to physical objects or surfaces in their physical environment. Enabling users to easily manipulate virtual objects to the desired position and orientation relative to physical objects or surfaces is challenging. Summary of the Invention

[0002] This paper discloses methods, computing devices, and head-mounted display devices for manipulating virtual objects displayed within a real-world physical environment. In one example, a method includes receiving image data of a real-world physical environment comprising multiple physical features. Using the image data, a 3D model of at least a portion of the real-world physical environment is generated. Multiple candidate anchor features are extracted from the image data, each candidate anchor feature corresponding to one of the physical features in the real-world physical environment.

[0003] The method includes receiving user input to manipulate a virtual object displayed in a real-world physical environment, wherein the virtual object includes virtual anchor features. Based on the manipulation of the virtual object, identifying at least one correspondence between the virtual anchor features of the virtual object and corresponding candidate anchor features from a plurality of candidate anchor features. Based on identifying at least one correspondence, displaying an indication of the corresponding candidate anchor feature at a corresponding physical feature in the real-world physical environment via a display device.

[0004] The Summary section is provided to introduce some concepts in a simplified form, which will be further described in the Detailed Description section below. The Summary section is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations capable of addressing any or all of the defects mentioned in any part of this disclosure. Attached Figure Description

[0005] Figure 1 This is a schematic diagram of an example computing device and display device that can display and manipulate virtual objects according to the examples of this disclosure.

[0006] Figure 2 The following is an example of wearing the device in a real-world physical environment according to this disclosure. Figure 1 Users of head-mounted display devices.

[0007] Figure 3 Examples of the present disclosure are shown. Figure 2 Examples of candidate anchor point features corresponding to physical features in a real-world physical environment.

[0008] Figures 4 to 7 The edge of a virtual cube aligned with the edge of a physical table is shown as an example according to this disclosure.

[0009] Figures 8 to 10 The illustration shows an indication of corresponding candidate anchor point features extending from an invisible location into the field of view of the display device, as well as a virtual object physically aligned with the corresponding anchor point, according to an example of this disclosure.

[0010] Figure 11 and Figure 12 A virtual object, shown as moving along with its corresponding physical object according to an example of this disclosure, is illustrated.

[0011] Figure 13 and Figure 14 An example according to this disclosure is shown of a virtual object that remains at the virtual object's position when the corresponding physical object it is aligned with moves.

[0012] Figures 15 to 19 The example shown illustrates the creation of a virtual box using indications of corresponding candidate anchor point features at corresponding physical features within a physical environment, according to an example of this disclosure.

[0013] Figure 20A , Figure 20B and Figure 20C This is a flowchart illustrating a method for manipulating a virtual object displayed via a display device, according to an example of this disclosure.

[0014] Figure 21 A computing system according to an embodiment of the present disclosure is shown. Detailed Implementation

[0015] In a real-world physical environment, a person physically manipulating and positioning a first object in relation to a second object can be guided by real-world phenomena. For example, a person stacking books on a table is aided by gravity aligning the books with the table's supporting surface and by the fact that the books cannot penetrate the table's surface.

[0016] In augmented reality environments, users can view a real-world physical environment that includes virtual content displayed within that physical environment. For example, and as described in more detail below, head-mounted display (HMD) devices can include perspective displays configured to visually enhance a view of a real-world three-dimensional environment through the display. In other examples, virtual content can be blended with real-world images of the user's real-world environment. For example, tablet computers, mobile communication devices, laptop computers, and similar devices can display virtual content blended with real-world images of the user's real-world environment. Other examples of displaying virtual content in a real-world physical environment can include projection-based schemes, such as projectors mounted in space or projectors worn on the user's body.

[0017] In augmented reality environments, virtual objects do not necessarily follow the same physical phenomena or rules as physical objects in the real world. Therefore, manipulating virtual content to align or coordinate with physical objects or surfaces in the user's physical environment can be challenging. For example, in augmented reality applications that support moving virtual content in six degrees of freedom (6DOF), manipulating and orienting virtual objects to precisely align them with physical objects in the real-world environment can be cumbersome. An example of this challenge might be attempting to manipulate a holographic flat painting to hang it flat on a real-world wall.

[0018] Additionally, in augmented reality environments, virtual content can be displayed from the user's perspective and within the context of the surrounding physical environment. In some examples, such as using an HMD device with a perspective display, a zoomed-out view of the entire physical environment may not be feasible. Similarly, a magnified view of the alignment details between virtual and physical objects may not be an option. These and other considerations may prevent users from easily and quickly aligning virtual content within a real-world physical environment.

[0019] This disclosure relates to methods, computing devices, and display devices for providing interactive augmented reality experiences that enable users to easily and quickly manipulate virtual content precisely within a real-world physical environment. A potential advantage of this disclosure is that users can easily position virtual objects within their environment by utilizing real-world physical objects, surfaces, and geometry around them.

[0020] Now for reference Figure 1This document provides a schematic diagram of an example implementation of a computing and display device for manipulating displayed virtual objects. In one example, computing device 10 is integrated into a head-mounted display (HMD) device 18. Computing device 10 may include a virtual object manipulation program 12, which includes instructions that can be stored in a mass storage device 16. Virtual object manipulation program 12 may be loaded into memory 20 and executed by processor 22 to perform one or more methods and processes described herein. References are provided below. Figure 1 Additional details regarding the components and computing aspects of computing device 10 are described in more detail.

[0021] HMD device 18 can create and display an augmented reality environment including virtual content to a first viewer 24. HMD device 18 may include a display program 28 that generates such virtual content for display via the HMD device. The virtual content may include one or more visual elements in the form of virtual objects 30, such as three-dimensional (3D) holographic objects and two-dimensional (2D) virtual images, which are generated and displayed as if located within the real-world physical environment 32 viewed through the device. In this way, HMD device 18 can create an augmented reality environment that allows the viewer to perceive such virtual objects 30 within the physical environment 32 surrounding the viewer. As discussed in more detail below, the physical environment 32 may include physical objects having physical features 34 such as edges, planes, etc.

[0022] In some examples, HMD device 18 may include at least a partially transparent display 36 supported in front of one or both of a user's eyes, thereby providing the user with a view of his or her surroundings. Any suitable display technology and configuration can be used to display images via the at least partially transparent display 36. For example, the at least partially transparent display 36 may be configured such that the wearer of HMD device 18 can view physically real-world objects in the physical environment through one or more partially transparent pixels that are displaying representations of virtual objects. The at least partially transparent display 36 may include image-generating elements, such as, for example, a transparent organic light-emitting diode (OLED) display.

[0023] As another example, HMD device 18 may include a light modulator on the edge of one or more at least partially transparent display panels. In this example, the panel can act as a light guide to deliver light from the light modulator to the wearer's eyes. Such a light guide allows the wearer to perceive virtual content located within the physical environment the wearer is viewing. In other examples, the display panel may utilize a liquid crystal on silicon (LCOS) display.

[0024] HMD device 18 may include various sensors and related systems that receive physical environment data from physical environment 32. For example, HMD device 18 may include a depth sensor system 38 that generates depth image data. Depth sensor system 38 may include one or more depth cameras that capture image data 26 from physical environment 32. In some examples, the depth cameras (multiple) may be infrared time-of-flight depth cameras. In other examples, the depth cameras (multiple) may take the form of structured light depth cameras. Any suitable depth tracking system and technology can be used.

[0025] In some examples, the HMD device 18 may include an optical sensor system 40 that utilizes at least one outward-facing sensor, such as an RGB camera, an IR sensor, or other optical sensor. The outward-facing sensor can capture image data 26 in the form of color, IR, or other light information from the physical environment 32. In some examples, such image data 26 can be used by the processor 22 to detect movement within the field of view of the HMD device 18, such as gesture-based input or other movements performed by the wearer (e.g., pinching a finger, closing a fist, pointing with a finger or hand, etc.), which indicate actions to be taken, selections of virtual objects displayed via the HMD device 18, or other user input.

[0026] Data from the optical sensor system 40 can also be used by the processor 22 to determine orientation / position and orientation data (e.g., from imaging environment features) that support the position / motion tracking of the HMD device 18 in real-world physics 32. Such data can also be used to identify surfaces of the physical environment 32 and / or measure one or more surface parameters of the physical environment 32.

[0027] HMD device 18 may also include a position sensor system 42, which includes one or more accelerometers, gyroscopes, inertial measurement units, head tracking systems, and / or other sensors for determining the position and / or orientation of the device. The relative position and / or orientation of HMD device 18 with respect to physical environment 32 can be evaluated so that virtual content can be accurately displayed in the desired real-world location with the desired orientation.

[0028] In some examples, a 6-DOF position sensor system can be used to display virtual content in a world-locked manner. World-locked virtual objects, such as holograms, appear fixed relative to real-world objects that can be viewed through the HMD device 18, allowing the wearer of the HMD device to move in the real-world physical environment while perceiving the virtual objects as remaining stationary in the physical environment with fixed positions and orientations.

[0029] In other examples, the HMD device 18 can operate in a body-locked display mode, in which one or more virtual objects can be displayed via the HMD device in a body-locked position. In the body-locked position, the holographic object appears fixed relative to the wearer of the HMD device 18, and the body-locked position of the holographic object appears movable relative to the real-world object.

[0030] HMD device 18 may also include a transducer system 44, which includes one or more actuators that convert electrical signals into another form of energy. In some examples, transducer system 44 may include one or more speakers for providing audio feedback to a viewer. In other examples, transducer system 44 may include one or more haptic transducers for generating and providing haptic feedback (such as vibration) to a viewer. HMD device 18 may also include a microphone system 46 and one or more microphones for receiving audio input from the physical environment.

[0031] The computing device 10 can receive eye-tracking data from the eye-tracking system 68 of the HMD device 18. In some examples, one or more inward-facing light sources and image sensors can collect image information for measuring the user's gaze parameters. Using this information, the processor 22 can execute instructions to determine the direction the user is gazing and / or the identification of the physical and / or virtual objects the user is gazing at. Using the eye-tracking data, the processor 22 can execute instructions to monitor the gaze position of an observer within the physical environment 32 relative to the physical features 34 and virtual content displayed within the physical environment. In other examples, any suitable eye-tracking technology can be used.

[0032] In some examples, a 3D model 72 of at least a portion of the physical environment 32 may be generated by the HMD device 18 and used to display and manipulate virtual objects 30 within the physical environment. The 3D model may include surface reconstruction information, which can be used to identify physical features 34 in the physical environment, such as objects and surfaces.

[0033] As described in more detail below, candidate anchor features 74 can be extracted from image data 26, where the candidate anchor features correspond to physical features 34 in the real-world physical environment 32. At least one correspondence can be identified between the virtual anchor features of the virtual object 30 and the corresponding candidate anchor features 74. Using this correspondence, the HMD device 18 can display an indication 76 of the corresponding candidate anchor feature at the corresponding physical feature within the physical environment. As described in the example use case below, displaying such an indication allows the user to easily and accurately locate the virtual object in the environment using the physical features of the physical environment.

[0034] Figure 1The example shown illustrates a computing device 10 integrated into an HMD device 18. It should be understood that in other examples, the computing device 10 may be a component separate from and communicatively coupled to the HMD device 18. Additionally, many types and configurations of HMD devices 18 with various form factors can be used, and these HMD devices 18 are within the scope of this disclosure. In some examples, one or more of the aforementioned sensor systems or other data collection systems may be located external to the HMD device 18.

[0035] Continue to refer to Figure 1 This illustrates an example of a computing device 80 that is physically separate from a display device 82. In this example, the computing device 80 may include a separate device or be integrated into a separate device, such as a set-top box, a game console, or other similar device that does not include an integrated display.

[0036] Computing device 80 can be operatively connected to display device 82 via a wired connection, or via a wireless connection through WiFi, Bluetooth, or any other suitable wireless communication protocol. For example, computing device 80 can be communicatively coupled to network 84. Network 84 can be in the form of a local area network (LAN), wide area network (WAN), wired network, wireless network, personal area network, or a combination thereof, and may include the Internet. See below for reference. Figure 1 Further details regarding the components and computing aspects of computing device 80 are described in more detail.

[0037] Similar to computing device 10, computing device 80 may include a virtual object manipulation program 12 that can be stored in mass storage device 86. The virtual object manipulation program 12 may be loaded into memory 88 and executed by processor 90 to implement one or more methods and processes described in more detail below.

[0038] Example display device 82 may include a camera 92 for capturing image data 26 of the physical environment 32 and a display system 94 for presenting visual content to a second viewer 96. In some examples, display device 82 may include one or more of the sensors, transducers, microphones, and eye-tracking systems of the HMD device 18 described above.

[0039] In other examples, computing device 80 may include or be integrated into display device 82, and may take the form of a tablet computer, laptop computer, mobile computing device, wall-mounted monitor, or other similar device with an integrated display. In some examples, one or more external cameras and other sensors (not shown) may capture data about the position or orientation of display device 82 and provide such data to computing device 80. For example, one or more externally mounted depth cameras, RGB cameras, and / or IR cameras may track the position and orientation of display device 82 and provide such data to computing device 80.

[0040] Now for reference Figures 2 to 19 A description of example use cases for this disclosure will be provided. Figure 2 This is a schematic diagram of a user 200 wearing an HMD device 18 and standing in a real-world physical environment of room 210. Room 210 includes multiple physical objects and surfaces, such as walls 214, 216, and 218, a wall-mounted monitor 220, a bed 222, a table 230, and a bookcase 240. Each physical object may include one or more physical features, such as linear edges, the midpoint of a linear edge or line, a planar surface, a plane parallel to and spaced from a planar surface, corners, and angles. For example, table 230 has multiple linear edges such as edges 232 and 234 and multiple planar surfaces such as a tabletop 236. In other examples, the physical features of a physical object may include one or more of the object's volume, orientation, and size.

[0041] These physical features can represent useful semantic information, such as supporting horizontal surfaces, floors, walls that divide space, etc. For example, some linear edges can distinguish an object from other objects behind it, or they can represent changes in the surface normals of an object. Other linear edges may be caused by visible changes in the object's color.

[0042] As described above, the HMD device 18 can use the image data 26, position and / or orientation data of the HMD device 18 to generate a 3D model of at least a portion of the room 210. The virtual object manipulation program 12 can be executed to extract a plurality of candidate anchor point features 74 from the image data 26, each candidate anchor point feature 74 corresponding to a physical feature in the room 210. For example, the candidate anchor point features determined from the image data may be in the form of selected 3D linear edges and planar surfaces of physical objects in the room 210.

[0043] Now for reference Figure 3In one example, candidate anchor features 74 in the form of 3D line segments are determined from image data 26. These 3D line segments are, for example, line segments 310, 320, and 330, and each candidate anchor feature 74 corresponds to a selected linear edge of a bookshelf 240 in room 210. In this example, fewer than all linear edges of the bookshelf 240 are represented by line segment candidate anchor features. As described in more detail below, these extracted line segments can be used to display indications of the candidate anchor features 74 corresponding to these line segments, such as... Figure 2 The virtual cylinder indicated is 254.

[0044] In one example of extracting candidate anchor point features 74 in the form of 3D line segments, depth data can be used to detect depth edges. Exponential filters can be used to temporarily smooth the depth data, such as:

[0045]

[0046] Where α = 0.3. The surface normal at each point in the depth image is calculated; these surface normals are used to detect surface normal edges. Additionally, color image data can be used to detect color edges.

[0047] The most prominent lines can be extracted from each depth image component, normal component, and / or color image component. For example, a Hough transform can be run on the edge points detected by the Canny edge detection technique. A variant of RANSAC can be used to extract the internal edge points along each Hough line to generate a more accurate line equation. Finally, each 2D line is divided into segments with an edge point density greater than a predetermined threshold. In some examples, the location of the depth edge can be slightly shifted to lie on the approximately closed object that initiated the edge.

[0048] Given a sufficient number of valid depth values ​​along the length of each segment, the 2D line segments extracted from all modalities are back-projected into the 3D space of the 3D model. RANSAC can then be used to fit the 3D linear segments using the 3D edge points.

[0049] Temporal noise in the location, orientation, and / or range of the extracted segments can be reduced by using a Kalman filter. Such filtering can enhance the consistency of these segments across consecutive image data frames. The extracted segments can be associated with existing edges based on the proximity and similarity of the line segment orientations. In some examples, the aforementioned process for extracting candidate anchor point features 74 in the form of three-dimensional line segments can be performed at 15Hz, 30Hz, 60Hz, or other suitable frequencies.

[0050] As explained in more detail below, indicators of the extracted segments (corresponding candidate anchor point features) can be displayed to the user to assist in aligning the virtual object with the physical object corresponding to the extracted segment. In other examples, indicators of other corresponding candidate anchor point features different from the extracted line segments can be displayed to the user to assist in aligning the virtual object with the physical object corresponding to the candidate anchor point features. Such other corresponding candidate anchor point features may include, for example, different geometric features such as planes, cylinders, spheres, surfaces, volumes, dimensions, repetition distances between objects, and the relative positions and orientations of objects.

[0051] In some examples, the indicator will not be displayed in the current frame unless it has already been displayed in at least a predetermined number of previous frames. For example, if the indicator has been displayed in at least five frames prior to the current frame, then the indicator will be displayed. In this way, the chance of false alarms due to user reliance on the edge extraction process can be minimized. Depending on the sampling frequency and other relevant factors, any other suitable number of previous frames can be used.

[0052] In some examples, the user's physical environment can change dynamically. For example, and refer to... Figure 2 Another person might enter room 210 and potentially pass between the HMD device 18 and the table 230, temporarily blocking the device from acquiring image data of the table. In other examples, objects might move within or be removed from room 210, the user 200 might change their position in the room, new objects might be brought into the room, and so on. In some examples, the user can move physical objects in room 210 to generate different and / or additional guides for manipulating virtual objects. Therefore, in some of these examples, image data containing specific candidate anchor point features may not be available.

[0053] More specifically, in some examples, once a candidate anchor feature is detected in an image data frame at a 3D location in 3D model 72, the candidate anchor feature may not be detected in subsequent frames of image data. However, such a particular candidate anchor feature may still be valuable to the user, such as in cases where the corresponding physical feature / object is temporarily occluded from the view or remains useful at a specific location of a linear edge.

[0054] To address such situations, and in some examples, a specific candidate anchor feature will be maintained at its initial 3D position in the 3D model until a predetermined time period expires after the initial time. This predetermined time period can be, for example, one second, two seconds, or any suitable time interval. In some examples where a specific candidate anchor feature is currently being used to align a virtual object, the predetermined time period can be extended until the virtual object's alignment is complete. At the end of the predetermined time period after the initial time, the candidate anchor feature can be removed from the model. In this way, situations such as short periods of occlusion of candidate anchor features from the view, conflicts between candidate anchor features due to user movement, movement of physical objects in the scene, and / or other changes in the physical environment can be addressed.

[0055] Continue to refer to Figure 3 In some examples, candidate anchor features 74, such as planar surfaces 340 and 350 corresponding to selected planar surfaces of table 230 in room 210, are determined from image data 26. In this example, less than all planar surfaces of table 230 are represented by planar surface candidate anchor features. As described in more detail below, these planar surface candidate anchor features can be used to display indications of candidate anchor features 74 corresponding to these surfaces, such as Figure 2 The virtual circles shown indicate 260 and 264.

[0056] In one example of extracting candidate anchor point features 74 in the form of a planar surface, such a surface can be detected using any suitable scene analysis method or technique. For example, surface reconstruction techniques, such as those using contour reconstruction models, can be utilized. A plane equation parameterized by its azimuth, elevation, and distance from the origin can be generated using depth image normals. The principal plane equation in the scene can be detected using the Hough transform on 3D depth points.

[0057] In some examples, a greedy heuristic can be used to associate scene points with those planes. Unassigned 3D points can be associated with candidate planes if they are located near the candidate planes, such as within 10 cm, and have a compatible normal direction, such as within 10° of the candidate plane's normal. In some examples, for candidate planes beyond a predetermined distance from the depth camera, the range of such compatible normal directions can be increased, such as to approximately 20°, to account for increased depth data noise. In some examples, the aforementioned process for extracting candidate anchor point features 74 in the form of 3D planar surfaces can be performed at 2 Hz, 4 Hz, 10 Hz, or other suitable frequencies.

[0058] Refer again Figure 2Candidate anchor features 74 extracted from image data 26 can be used to help user 200 manipulate virtual content displayed by HMD device 18 within room 210. In one example, HMD device 18 can display virtual objects in the form of a 3D holographic cube 270 floating in room 210. As described in more detail below, holographic cube 270 may include one or more virtual anchor features that can be used to identify and indicate to the user corresponding candidate anchor features on physical objects in the room. In some examples, virtual anchor features of virtual objects such as holographic cube 270 may include linear edges and planar surfaces. Figure 2 In some examples of virtual objects such as the holographic tank 284, the virtual anchor point features of the object can be defined by a bounding box, such as bounding box 286, to facilitate alignment with the physical edges and planar surfaces (corresponding anchor point features) of the physical object.

[0059] In some examples, the HMD device 18 enables user 200 to interact with the virtual cube 270 via gaze detection of the user's gaze cube. In other examples, such interaction can be achieved by detecting a ray 274 projected from the HMD device and intersecting with cube 270, by receiving a user voice command to select a cube for interaction, or by any other suitable user interaction technology. For the purposes of the examples described herein, various forms of user input can be provided by one or more of gaze detection, ray projection, head position / orientation detection, voice command recognition, gesture detection, handheld remote control input, and any other suitable technology for providing user input.

[0060] User 200 can provide user input to HMD device 18 to manipulate the holographic cube 270. In some examples, the user can position, orient, and scale the cube 270 via the user's head gaze direction and / or other forms of user input. In one example, by changing his / her head gaze direction, user 200 can move the cube 270 within room 210. The distance between the cube 270 and HMD device 18 can be controlled via user voice commands, such as "backward" and "forward". In another example of scaling the cube 270, user 200 can deselect the cube, look at a corner of the cube, issue a "scale it" command, and move his / her gaze away from the corner to uniformly increase the size of the cube.

[0061] In some examples, user 200 might want to align the linear edges or planar surfaces of the holographic cube 270 with the edges or surfaces of physical objects in room 210, such as table 230. Now refer to... Figures 4 to 7In one example, user 200 might want to align the bottom linear edge 278 of cube 270 with the top linear edge 280 of table 230. For example... Figure 4 As shown, initially, user 200 can manipulate cube 270 by, for example, selecting the cube for user interaction and moving it upwards in the direction of arrow 400. In other examples, the user can manipulate the virtual object by changing its orientation, modifying its shape, or otherwise interacting with it.

[0062] HMD device 18 can receive user manipulation input from user 200. Based on the user's manipulation of cube 270, virtual object manipulation program 12 can identify at least one correspondence between virtual anchor features of the virtual object (cube 270) and corresponding candidate anchor features from a plurality of candidate anchor features in room 210. In this example, each of the 12 linear edges and 6 rectangular planar surfaces of holographic cube 270 is a virtual anchor feature of the cube. As explained above, HMD device 18 extracts a plurality of candidate anchor features, each corresponding to a physical feature of a physical object in room 210. In this example, each of the planar surface of desktop 236 and each of the four top linear edges 280, 288, 290, and 292 is a candidate anchor feature.

[0063] When user 200 moves cube 270 upwards in the direction of arrow 400, virtual object manipulation program 12 can identify the correspondence between the top surface 294 of cube 270 and the planar surface of desktop 236 (corresponding to candidate anchor point features). In one example, the correspondence may include the surface normal of desktop 236 and the top surface 294 of cube 270 being within a predetermined parallel threshold, such as 30 degrees, 45 degrees, 60 degrees, or any suitable threshold. In other examples, the correspondence may include the surface normal of desktop 236 and the top surface 294 of cube 270 being within a predetermined orthogonality threshold, such as 30 degrees, 45 degrees, 60 degrees, or any suitable threshold.

[0064] In another example, the correspondence could include movement of the top surface 294 of cube 270 toward the tabletop 236, indicating a potential user interest in the alignment of the two surfaces. In this example, the virtual object manipulator 12 could analyze the top surface 294 and the tabletop 236 in the context of such movement to generate inferences about the user's potential interest in aligning the two surfaces. In another example, the correspondence could include a combination of surface normals within a predetermined parallel threshold and movement of the top surface of cube 270 toward the tabletop 236.

[0065] In some examples of identifying multiple correspondences with multiple corresponding candidate anchor features, the corresponding candidate anchor feature with the lowest cost according to the cost function can be utilized. In one example, the cost of the corresponding candidate anchor feature is a linear combination of the following: (a) the distance between the centroid of the virtual anchor feature and its projection onto the corresponding candidate anchor feature, and (b) the distance between the centroid of the virtual anchor feature and the centroid of the corresponding candidate anchor feature. In some examples, one or both of distances (a) and (b) can be weighted to increase or decrease the cost. In this example, the second component (b) of this linear combination favors the planar surface that is close to the virtual anchor feature surface.

[0066] In some examples, a corresponding candidate anchor feature previously selected for use with a virtual anchor feature may be preferred over a candidate anchor feature that has not been previously selected or has been rarely selected. For example, if user 200 places a virtual object on desktop 236 twice as frequently as any other planar surface in room 210, the corresponding candidate anchor feature for desktop 236 may be selected over other planar surfaces. In some examples, the frequency with which other users select desktop 236 may also be considered favorable or unfavorable to that candidate anchor feature. In this way, candidate anchor features that are more popular with users can be identified and utilized more frequently. Additionally, other examples that may suggest potential user interests in aligning virtual anchor features with corresponding candidate anchor features may be utilized, and these examples are within the scope of this disclosure.

[0067] In some examples, when the correspondence is identified, cube 270 can be rotated and / or translated such that its upper surface 294 is parallel to and coplanar with the corresponding candidate anchor point feature plane surface of tabletop 236. In other examples, cube 270 can be rotated and / or translated such that its upper surface 294 is orthogonal to the corresponding candidate anchor point feature plane surface of tabletop 236.

[0068] Continue to refer to Figure 4 Based on the correspondence between the upper surface 294 of the identifier cube 270 and the corresponding candidate anchor point features of the planar surface of the desktop 236, the HMD device 18 displays to the user 200 an indication of the corresponding candidate anchor point feature at the corresponding physical feature (desktop surface). Figure 4In this example, the indication displayed by HMD device 18 is in the form of a virtual circular area 410 coplanar with and located at the centroid of desktop 236. In this example, the indication also includes a linear segment 414 extending upwards from the center of the circular area 410, which is perpendicular to the desktop plane. The virtual circular area 410 and the linear segment 414 can be displayed in a world-locked manner, where their position and orientation appear fixed relative to the table 230 and other physical objects in the room 212. In this way, a visual indication is provided to user 200 highlighting the planar nature of desktop 236, thus informing the user that alignment with this physical surface is feasible.

[0069] exist Figure 4 In the example, the indication of the corresponding candidate anchor point feature for the desktop 236 also includes an extension 420, which is shown to extend from the center of a circular region parallel to the planar desktop 236 and beyond the physical desktop 236. In this example, the direction of the extension 420 is chosen based on the direction of movement of the holographic cube 270 (indicated by arrow 400) such that continued movement of the cube in that direction will cause the cube to intersect with the extension 420. Therefore, by positioning the extension 420 to extend above the holographic cube 270 when viewed from the y-axis direction, the user 200 can easily coordinate the position of the cube relative to the planar surface of the desktop 236. In other examples, indications with different shapes and forms can be used to indicate the corresponding candidate anchor point feature representing the planar surface.

[0070] Now for reference Figure 5 and Figure 6 User 200 can raise cube 270 until its bottom surface meets extension 420. The user can then begin moving cube 270 along the x-axis direction indicated by arrow 430 toward the top linear edge 280 of table 230, which, as described above, is represented by the candidate anchor point features of the table. As the user moves cube 270 in the direction of arrow 430, virtual object manipulation program 12 can identify a correspondence between the bottom linear edge 278 of cube 270 and the corresponding candidate anchor point features of the top linear edge 280 of the table. In one example, the correspondence may include the bottom linear edge 278 and the top linear edge 280 within a predetermined parallel threshold, such as 10 degrees, 20 degrees, 30 degrees, or any suitable threshold. In another example, the correspondence may include movement of the bottom linear edge 278 of cube 270 toward the top linear edge 280 of the table to indicate to the user a potential interest in aligning the two edges. In another example, the correspondence may include a combination of these edges within a predetermined parallel threshold and the movement of the bottom linear edge 278 toward the top linear edge 280.

[0071] In another example, the correspondence may include the distance between the midpoint of the bottom linear edge 278 and its projection onto the top linear edge 280 being within a predetermined threshold distance (such as 10cm, 20cm, 30cm or other suitable distance) and the bottom linear edge and the top linear edge being within a predetermined parallel threshold, such as within 10 degrees, 20 degrees, 30 degrees or any suitable threshold.

[0072] As noted above regarding planar surfaces, in some examples of multiple correspondences between identifying and representing multiple corresponding candidate anchor features of linear edges, the corresponding candidate anchor features with the lowest cost can be utilized, where the cost is a linear combination of (a) the distance between the midpoint of the virtual anchor feature edge and its projection onto the physical edge of the corresponding candidate anchor feature, and (b) the distance between the midpoint of the virtual anchor feature edge and the midpoint of the edge of the corresponding candidate anchor feature. In some examples, one or both of distances (a) and (b) can be weighted to increase or decrease the cost. In this example, the second component (b) of the linear combination favors edges closer to the virtual anchor feature edge.

[0073] As described above and in some examples, corresponding candidate anchor feature edges that have been previously selected for use with virtual anchor features may be superior to candidate anchor feature edges that have not been previously selected or have been rarely selected previously. Additionally, other examples that may suggest potential user interests in aligning virtual anchor features with corresponding candidate anchor features can be utilized, and these examples are within the scope of this disclosure.

[0074] In some examples, when the correspondence is identified, the cube 270 can be rotated and / or translated so that its bottom linear edge 278 is parallel to and overlaps with the corresponding candidate anchor feature edge.

[0075] Continue to refer to Figure 5 and Figure 6 Based on the correspondence between the bottom linear edge 278 of the identifier cube 270 and the corresponding candidate anchor point features of the top linear edge 280 of the table, the HMD device 18 can display indications of the corresponding candidate anchor point features at corresponding physical features in the real-world physical environment. Figure 5 and Figure 6In the example, the indication displayed by HMD device 18 is in the form of a virtual cylinder 510 extending along the edge 280 of table 230. In this example, the virtual cylinder 510 has a substantially the same length as the edge 280 and is coaxial with the edge 280. The virtual cylinder 510 can have a degree of transparency, allowing the edge 280 to remain visible to user 200, or it can be opaque. The virtual cylinder 510 can be displayed in a world-locked manner, making its position and orientation appear fixed relative to table 230 and other physical objects in room 212. In this way, user 200 is provided with a visual indication that the top linear edge 280 is highlighted, thus letting the user know that alignment with that physical edge is feasible.

[0076] exist Figure 5 and Figure 6 In one example, the indication of the corresponding candidate anchor point feature representing the top linear edge 280 also includes an extension 520, which is shown extending from the center of the virtual cylinder 510, collinear with the axis of the cylinder, and beyond each end of the edge 280. In this way, the virtual object manipulation procedure 12 allows the user 200 to clearly see the position and orientation of the table edge 280. In some examples, and as described in more detail below, displaying such an extension helps the user 200 align the virtual object with candidate anchor point features that are not visible or difficult to see for the user, such as physical edges located behind or far from the user. In other examples, indications with different shapes and forms can be used to indicate the corresponding candidate anchor point feature representing the linear edge.

[0077] Now for reference Figure 7 User 200 can provide user selection input to align the virtual bottom linear edge 278 of cube 270 with the physical table edge 280. User selection input can include one or more of the following: gaze detection, raycasting, head position / orientation, voice commands, gesture detection, handheld remote control input, and any other suitable techniques for providing user input. In this example, aligning the virtual bottom linear edge 278 with the physical table edge 280 involves rotating and translating cube 270 to position the virtual edge parallel to and adjacent to the physical edge, such that the two edges appear to touch, as... Figure 7 As shown. In other examples, aligning the virtual anchor feature edge with the corresponding candidate anchor feature edge can include orienting the virtual object such that the virtual edge is parallel to and spaced apart from the physical edge.

[0078] Refer again Figure 2In some examples, the complexity of real-world geometry may produce a dense and noisy set of candidate anchor features that can be extracted. Sensor noise from one or more sensors of the HMD device 18 may also result in high-density candidate anchor features. In some examples, high-density candidate anchor features may adversely affect a user's ability to effectively utilize real-world objects, surfaces, and geometry to manipulate virtual objects as described herein.

[0079] In some examples, the virtual object manipulation procedure 12 allows the user to actively pre-select one or more candidate anchor features for extraction. In this way, and by placing the user within a loop of the candidate anchor feature extraction process, the effects of real-world geometric complexity and sensor noise can be reduced. In one example, and again referring to... Figure 2 User 200 can preselect the top linear edge 280 of table 230 by providing user preselection input to HMD device 18. Such user preselection input can take one or more of the following forms: gaze detection, ray projection, head position / orientation, voice command, gesture detection, handheld remote control input, and any other suitable technology for providing user input.

[0080] When the user manipulates the virtual cube 270 around the table 230, the virtual object manipulation program 12 can also utilize this user pre-selection of the linear edge 280 to identify the correspondence between the virtual anchor feature edge 278 and the corresponding candidate anchor feature of the linear edge 280. More specifically, and based on this pre-selection input, the corresponding candidate anchor feature of the linear edge 280 can be selected over other candidate anchor features corresponding to other physical features of other physical objects in the table 230 and room 210. In some examples, and based on this pre-selection input, the indication of other candidate anchor features can be not shown to the user 200, while the indication of the pre-selected linear edge 280 is shown, such as... Figure 5 and Figure 6 As shown.

[0081] In some examples, users can deselect one or more corresponding candidate anchor features. By providing such user deselection input, the corresponding candidate anchor features that have been deselected can be left unselected, and indications of such candidate anchor features can be hidden. This user deselection input can also reduce the density of the extracted candidate anchor features.

[0082] In some examples, the potential negative impact of complex real-world geometry and / or sensor noise can be reduced by limiting the types (and corresponding numbers) of extracted candidate anchor features. In one example, less than all available virtual anchor features of a virtual object can be utilized. In other words, a virtual object can be defined to exclude one or more virtual anchor features from being used for alignment.

[0083] For example, the virtual cube 270 has six planar surfaces. To limit the number of candidate anchor point features extracted, each planar surface of the cube can be constrained to align only with physical planar surfaces, and correspondingly, it can be prevented from aligning with physical corners or edges. Therefore, the extraction process for extracting candidate anchor point features from image data can ignore features corresponding to such physical corners and edges in the physical environment. In this way, fewer candidate anchor point features can be extracted, thereby reducing the density of potential candidate anchor point features indicated and / or utilized for alignment, and also potentially reducing the computational resources required for the extraction process.

[0084] In some examples, the potential negative impact of complex real-world geometry and / or sensor noise can be reduced by generating multiple filtered candidate anchor features based on one or more criteria. For example, multiple candidate anchor features available in room 210 can be filtered based on the distance between HMD device 18 and each candidate anchor feature. In some examples, a candidate anchor feature located closer to HMD device 18 may be superior to another candidate anchor feature located farther from the display. In another example, the distance between a virtual object such as cube 270 and each candidate anchor feature in room 210 can be utilized. For example, a candidate anchor feature located closer to cube 270 may be superior to another candidate anchor feature located farther from the cube.

[0085] In some examples, candidate anchor features can be prioritized based on whether they are within the field of view of the display device. For example, candidate anchor features not within the field of view of the HMD device 18 may be unavailable for selection, or may be less favored than other candidate anchor features within the field of view. In some examples, the prior usage frequency of each candidate anchor feature can be used to prioritize these features. For example, the priority of a candidate anchor feature may increase or decrease proportionally to its prior usage frequency when aligning with virtual objects. In some examples, candidate anchor features can be prioritized based on user-provided filtering input. For example, speech recognition can be used to recognize and execute voice commands from the user for filtering such features, such as "align to features only from the bookshelf."

[0086] In some examples, at least one correspondence between virtual anchor features of a virtual object and corresponding candidate anchor features can be identified from multiple candidate anchor features filtered according to one or more of the criteria described above. In this way, and as described above, fewer candidate anchor features can be extracted, thereby reducing the density of potential candidate anchor features indicated and / or utilized for alignment, and potentially reducing the computational resources required for the extraction process.

[0087] In some examples, a physical object may include one or more candidate anchor point features, which may be located in a real-world physical environment at an invisible location outside the field of view of the display device. For example and reference Figure 8 User 200 might wish to place a virtual baseball in a virtual display box 710 on the top flat surface 296 of the bookshelf 240. For example... Figure 8 As shown, because user 200 is looking away from bookshelf 240, the top plane surface 296 is out of view of the user and outside the field of view 720 of HMD device 18. User 200 can select box 710, such as by pointing to the box, and can begin moving the box upwards in the direction of arrow 730.

[0088] Candidate anchor point features of the bookshelf 240 may have been extracted from image data collected by the HMD device 18 while the user 200 was viewing the area of ​​the bookshelf. Therefore, when the user 200 moves the virtual display box 710 upwards, even though the bookshelf 240 is outside the field of view 720 of the HMD device 18, the correspondence between the bottom plane 740 of the box (virtual anchor point feature) and the top plane surface 296 of the bookshelf 240 (corresponding candidate anchor point feature) is identified. Based on the identified correspondence, an indicator 750 is generated and positioned on the top plane surface 296 in the 3D model. To inform the user 200 that the top plane surface 296 is behind him, an extension 760 of the indicator 750 is displayed extending from an invisible position on the top plane surface into the field of view 720 of the HMD device.

[0089] like Figure 8 and Figure 9 As shown, user 200 can use the shown extension 760 to position the bottom plane 740 of box 710 as coplanar with the top plane surface 296 of bookcase 240, even if the bookcase is behind the user and outside the user's field of vision.

[0090] In some examples, the user can pre-select the physical object and / or physical features of the object, as described above, such that the corresponding candidate anchor features are selected and the indication of the candidate anchor features is displayed to the user, even if the physical object and candidate anchor features are outside the field of view of the HMD device 18.

[0091] refer to Figure 9 and Figure 10 User 200 can provide user selection input that causes box 710 to translate, rotate, and / or reorient to be positioned on top of bookcase 240, such that the bottom plane 740 of the box appears to rest on the top plane surface 296 of the bookcase (see [link]). Figure 10In this way, when user 200 turns to bring the bookshelf into the field of view of HMD device 18, user 200 will perceive that box 710 is located on top of bookshelf 240.

[0092] Continue to refer to Figure 10 In some examples, when the virtual anchor point feature of a virtual object aligns with the corresponding candidate anchor point feature of a corresponding physical feature located in an invisible position, the HMD device 18 can output a directional notification. In this way, the HMD device 18 can warn the user 200 that the virtual object has been aligned to an invisible position.

[0093] In one example and as Figure 10 As shown, when the bottom plane 740 of the virtual box 710 is aligned with the top plane surface 296 of the bookshelf 240, the directional speaker of the HMD device 18 can broadcast a directional sound "ding-dong," which is perceived by the user 200 as coming from behind and above him at approximately the location of the virtual box 710. It can be understood that in... Figure 10 In the example, the alignment of the virtual box 710 with the top plane surface 296 of the bookshelf 240 can be achieved in a 3D model of the room 210. In this way, when the user 200 turns to bring the bookshelf into the field of view of the HMD device 18, the box 710 will be displayed as if it is on top of the bookshelf 240.

[0094] It should be understood that other types and forms of directional notification can be used to transmit 2D or 3D positions in the real-world physical environment to the user 200. For example, directional notification may include haptic output from the transducer of HMD device 18, such as one or more vibrations or vibration patterns. In some examples, the position of haptic feedback on HMD device 18 may correspond to the position of a virtual object aligned with a physical object in the real-world physical environment. In other examples where the user provides input via a handheld controller, the controller may generate sound or vibration to signal that a virtual anchor feature of the virtual object has been aligned with a corresponding candidate anchor feature of the corresponding physical feature.

[0095] When a virtual object is aligned with a physical object, the virtual object can have virtual anchor point features aligned with the corresponding candidate anchor point features of the physical object's physical characteristics. For example, and referencing... Figure 11 The enclosure 286 of the holographic tank 284 is displayed on the tabletop 236 of the table 230, such that the bottom plane of the cube is aligned with the top plane of the tabletop.

[0096] like Figure 12As shown, when the table 230 moves away from the bed 222, the bounding box 286 and the tank 284 move accordingly, such that the bottom plane of the cube (virtual anchor point feature) remains aligned with the top plane surface 236 (corresponding candidate anchor point feature of the corresponding physical feature). In this way, when the table 230 moves, the holographic tank 284 can appear to remain on the tabletop 236 in a natural manner.

[0097] In another example and reference Figure 13 The bounding box 286 and the holographic tank 284 can be displayed on the desktop 236 at positions corresponding to the virtual object positions 800 in the 3D model of room 210. In this example, the box 286 and the tank 284 can be fixed to this virtual object position 800 in the model. Therefore, and referring to... Figure 14 When table 230 moves to a new position within room 210, box 286 and tank 284 remain displayed at the fixed virtual object position 800.

[0098] In some examples, virtual objects can be created with the assistance of one or more techniques disclosed here. See now for reference. Figure 15 User 200 (not shown) can gaze at a point on desktop 236, where the gaze detection system of HMD device 18 determines the intersection point 810 of the user's gaze direction and the plane of the desktop. In this example, cursor 820 is displayed at the gaze intersection point 810, where the cursor is controlled by the user via user input such as gaze direction. The detected planar surface of desktop 236 can be indicated by a virtual circular area 830 and a surface normal 834.

[0099] refer to Figure 16 The user can provide drawing input, such as via voice commands, enabling the user to draw a three-dimensional edge 840 on the plane of desktop 306, starting from intersection 810. The endpoint of the edge, indicated by cursor 820, can be controlled by the user's gaze direction, and the edge 840 can be aligned with nearby physical linear edges. Such alignment can be expressed as extending the endpoint of edge 840 to a nearby physical linear edge (such as the right edge of the desktop 844). Such alignment can also be expressed as orienting edge 840 parallel or perpendicular to a physical linear edge. Figure 16 As shown, in this example, the indicator 850 (corresponding to the candidate anchor point feature) of the right edge of the table 844 can be displayed as described above.

[0100] Now for reference Figure 17The user can provide another user input for the endpoint of the fixed edge 840, and by moving their gaze direction, can begin to extend the rectangular region 854 away from the fixed edge. The user can determine the size of the rectangular region 854 by providing another user input for the size of the fixed region. Figure 17 As shown, in this example, the indicator 860 (corresponding to the candidate anchor point feature) near the edge 864 of the table can be displayed as described above.

[0101] Now for reference Figure 18 The 3D box 870 can protrude upwards from the desktop 236. In some examples, the height of the 3D box can be determined by finding the nearest intersection between the user's head direction and the light projection from the HMD device along the protruding direction of the 3D box, starting from the center of the box. The height of the box 870 can also be aligned with the physical plane surface and / or nearby edge endpoints. Figure 18 In the example, in response to the upward protrusion of the three-dimensional box 870, an extension 874 of the indicator 878 corresponding to the top planar surface 296 of the bookcase 240 can be displayed. In this way and with reference to... Figure 19 Users can easily align the top flat surface 880 of box 870 with the top flat surface 296 of bookcase 240 so that these surfaces are coplanar.

[0102] Figure 20A , Figure 20B and Figure 20C A flowchart illustrating a method 900 for manipulating a virtual object displayed via a display device, according to an example of this disclosure, is shown. (Refer to the above...) Figures 1 to 19 The software and hardware components described and illustrated herein are used to provide the following description of method 900. It should be understood that method 900 may also be performed in other contexts using other suitable hardware and software components.

[0103] refer to Figure 20A At 904, method 900 may include receiving image data of a real-world physical environment comprising multiple physical features. At 908, method 900 may include using the image data to generate a 3D model of at least a portion of the real-world physical environment. At 912, method 900 may include extracting multiple candidate anchor features from the image data, each candidate anchor feature corresponding to one of the physical features in the real-world physical environment. At 916, method 900 may include receiving user manipulation input to manipulate a virtual object displayed within the real-world physical environment, wherein the virtual object includes virtual anchor features.

[0104] At 920, method 900 may include: identifying at least one correspondence between a virtual anchor feature of the virtual object and a corresponding candidate anchor feature from a plurality of candidate anchor features, based on manipulation of the virtual object. At 924, method 900 may include, based on identifying at least one correspondence, displaying an indication of the corresponding candidate anchor feature at a corresponding physical feature in a real-world physical environment via a display device. At 928, method 900 may include aligning the virtual anchor feature of the virtual object with the corresponding candidate anchor feature of the corresponding physical feature based on receiving user selection input. At 932, method 900 may include a virtual object displaying the alignment of its virtual anchor feature with the corresponding candidate anchor feature.

[0105] Now for reference Figure 20B At 936, method 900 may include a physical feature found on a physical object in a real-world physical environment, and the method further includes displaying a corresponding movement of a virtual object as the physical object moves within the real-world physical environment, such that the virtual anchor point feature of the virtual object remains aligned with the corresponding candidate anchor point feature of the corresponding physical feature on the physical object. At 940, method 900 may include a physical feature found on a physical object in a real-world physical environment, and a virtual object displayed at a virtual object location in the real-world physical environment, and the method further includes maintaining the display of the virtual object at the virtual object location as the physical object moves within the real-world physical environment.

[0106] At 944, method 900 may include multiple physical features including three-dimensional linear edges and three-dimensional planar surfaces. At 948, method 900 may include displaying an indication extending from the invisible location into the field of view of the display device, where the corresponding physical feature is located in an invisible location outside the field of view of the display device in the real-world physical environment. At 952, method 900 may include outputting a directional notification indicating the invisible location of the corresponding physical feature when a virtual anchor point feature of a virtual object is aligned with a corresponding candidate anchor point feature of the corresponding physical feature located in the invisible location.

[0107] At 956, method 900 may include displaying an indication of a corresponding candidate anchor feature as extending beyond the corresponding physical feature. At 960, method 900 may include generating a filtered plurality of candidate anchor features by filtering a plurality of candidate anchor features based on one or more of the following: (1) the distance between the display device and each of the candidate anchor features, (2) the distance between a virtual object and each of the candidate anchor features, (3) whether each of the candidate anchor features is within the field of view of the display device, (4) the frequency of previous use of each of the candidate anchor features; and (5) user-filtered input.

[0108] Now for reference Figure 20C At 964, method 900 may include identifying at least one correspondence from a plurality of filtered candidate anchor features. At 968, method 900 may include receiving user pre-selection input for selected physical features from a plurality of physical features, wherein the selected physical features include corresponding physical features of the candidate anchor features, and wherein identifying at least one correspondence is also based on the user pre-selection input for the selected physical features. At 972, method 900 may include determining that the corresponding candidate anchor features are located at an initial 3D position in the 3D model and are extracted from frames of image data received at an initial time, determining that the corresponding candidate anchor features do not exist in subsequent frames of image data received after the initial time, and maintaining the corresponding candidate anchor features at the initial 3D position in the 3D model until a predetermined time period after the initial time expires.

[0109] It should be understood that method 900 is provided as an example and is not intended to be restrictive. Therefore, it should be understood that method 900 can include methods similar to... Figure 20A , Figure 20B and Figure 20C The steps shown may include more and / or alternative steps compared to those shown. Furthermore, it should be understood that method 900 can be performed in any suitable order. Additionally, it should be understood that one or more steps may be omitted from method 900 without departing from the scope of this disclosure.

[0110] In some embodiments, the methods and processes described herein can be attached to a computing system of one or more computing devices. In particular, such methods and processes can be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0111] Figure 21 The illustrations depict non-limiting embodiments of a computing system 1000 that can implement one or more of the methods and processes described above. Figure 1 The computing devices 10 and 80 shown may take the form of or include one or more aspects of computing system 1000. Computing system 1000 is shown in a simplified form. It should be understood that any computer architecture can be used practically without departing from the scope of this disclosure. In various examples, computing system 1000 may take the form of, or be communicatively coupled to, a head-mounted display device, tablet computer, home entertainment computer, desktop computer, network computing device, tablet PC, laptop, smartphone, gaming device, other mobile computing device, etc.

[0112] The computing system 1000 includes a logic processor 1004, volatile memory 1008, and non-volatile storage device 1012. The computing system 1000 may optionally include a display subsystem 1016, an input subsystem 1020, a communication subsystem 1024, and / or... Figure 21 Other components not shown.

[0113] The logic processor 1004 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transition the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0114] A logic processor may include one or more physical processors (hardware) configured to execute software instructions. Alternatively or additionally, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of logic processor 1004 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic processor may optionally be distributed across two or more separate devices that may be remotely located and / or configured for collaborative processing. Aspects of the logic processor may be virtualized and executed by remotely accessible, networked computing devices configured for cloud computing. In this case, these virtualized aspects may run on different physical logic processors on various different machines.

[0115] Volatile memory 1008 may include a physical device having random access memory. Volatile memory 1008 is typically used by logic processor 1004 to temporarily store information during the processing of software instructions. It should be understood that when the power supply to volatile memory 1008 is cut off, volatile memory 1008 typically does not continue storing instructions.

[0116] The non-volatile storage device 1012 includes one or more physical devices configured to store instructions executable by a logic processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 1012 can be changed—for example, to store different data.

[0117] The non-volatile storage device 1012 may include removable and / or built-in physical devices. The non-volatile storage device 1012 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, magnetic tape drive, MRAM, etc.) or other high-capacity storage technologies. The non-volatile storage device 1012 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that the non-volatile storage device 1012 is configured to retain instructions even when the power supply to the non-volatile storage device 1012 is cut off.

[0118] The logic processor 1004, volatile memory 1008, and non-volatile storage device 1012 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), programmable and application-specific integrated circuits (PASICs / ASICs), programmable and application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SOCs), and complex programmable logic devices (CPLDs).

[0119] The term "program" can be used to describe various aspects of a computing system 1000 implemented to perform a specific function. In some cases, a program can be instantiated using a portion of volatile memory 1008, via logic processor 1004, executing instructions stored in non-volatile memory 1012. It is understood that different programs can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same program can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The term "program" includes individual or group executable files, data files, libraries, drivers, scripts, database records, etc.

[0120] When included, the display subsystem 1016 can be used to present a visual representation of data stored by the non-volatile storage device 1012. As the methods and processes described herein change the data stored by the non-volatile storage device, and thus transform the state of the non-volatile storage device, the state of the display subsystem 1016 can also be transformed to visually represent the changes in the underlying data. The display subsystem 1016 may include one or more display devices utilizing virtually any type of technology. Such a display device may be combined with a logic processor 1004, volatile memory 1008, and / or non-volatile storage device 1012 in a shared package, or such a display device may be an external display device. Regarding Figure 1Example HMD device 18, example display subsystem 1016 is a perspective display 36 configured to visually enhance the appearance of a real-world three-dimensional physical environment by displaying virtual objects such as holograms.

[0121] When included, the input subsystem 1020 may include or interface with one or more user input devices. In some embodiments, the input subsystem may include or interface with a selected Natural User Input (NUI) component. Such a component may be integrated or external, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include a microphone for speech and / or voice recognition; an infrared, color, stereo, and / or depth camera for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, inertial measurement unit, and / or gyroscope for motion detection, gaze detection, and / or intent recognition; an electric field sensing component for assessing brain activity; any sensors described above with respect to HMD device 18, and / or any other suitable sensors.

[0122] When included, the communication subsystem 1024 can be configured to communicatively couple the computing system 1000 to one or more other computing devices. The communication subsystem 1024 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem may allow the computing system 1000 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0123] The following paragraphs provide additional support for the claims of this application. One aspect provides a method for manipulating a virtual object displayed via a display device, the method comprising: receiving image data of a real-world physical environment including a plurality of physical features; using the image data to generate a three-dimensional model of at least a portion of the real-world physical environment; extracting a plurality of candidate anchor features from the image data, each candidate anchor feature corresponding to a physical feature in the real-world physical environment; receiving user manipulation input to manipulate the virtual object displayed within the real-world physical environment, wherein the virtual object includes virtual anchor features; identifying at least one correspondence between the virtual anchor features of the virtual object and corresponding candidate anchor features from the plurality of candidate anchor features based on the manipulation of the virtual object; and displaying an indication of the corresponding candidate anchor feature at the corresponding physical feature in the real-world physical environment via the display device based on the identification of at least one correspondence. The method may additionally or optionally include: aligning the virtual anchor features of the virtual object with the corresponding candidate anchor features of the corresponding physical feature based on received user selection input; and displaying the virtual object with the virtual anchor features aligned with the corresponding candidate anchor features. The method may additionally or optionally include: wherein the corresponding physical feature is found on a physical object in a real-world physical environment, and when the physical object moves in the real-world physical environment, the corresponding movement of the virtual object is displayed so that its virtual anchor point feature remains aligned with the corresponding candidate anchor point feature of the corresponding physical feature on the physical object. The method may additionally or optionally include: wherein the corresponding physical feature is found on a physical object in a real-world physical environment, and the virtual object is displayed at the virtual object's location in the real-world physical environment, and when the physical object moves in the real-world physical environment, the display of the virtual object at the virtual object's location is maintained. The method may additionally or optionally include: wherein the plurality of physical features includes three-dimensional linear edges and three-dimensional planar surfaces. The method may additionally or optionally include: wherein the corresponding physical feature is located at an invisible location in the real-world physical environment outside the field of view of the display device, and an indication extending from the invisible location into the field of view of the display device is displayed. The method may additionally or optionally include: when the virtual anchor point feature of the virtual object is aligned with the corresponding candidate anchor point feature of the corresponding physical feature located at the invisible location, a directional notification indicating the invisible location of the corresponding physical feature is output. The method may additionally or optionally include: wherein the indication of the corresponding candidate anchor point feature is displayed as extending beyond the corresponding physical feature.The method may additionally or optionally include: generating a plurality of filtered candidate anchor features by filtering a plurality of candidate anchor features based on one or more of the following: (1) the distance between the display device and each of the candidate anchor features, (2) the distance between the virtual object and each of the candidate anchor features, (3) whether each candidate anchor feature is within the field of view of the display device, (4) the frequency of previous use of each of the candidate anchor features, and (5) user filtering input. The method may additionally or optionally include: wherein identifying at least one correspondence further includes identifying at least one correspondence from the plurality of filtered candidate anchor features. The method may additionally or optionally include: receiving user pre-selection input for selected physical features from a plurality of physical features, wherein the selected physical features include corresponding physical features of the candidate anchor features, and wherein identifying at least one correspondence is also based on the user pre-selection input for the selected physical features. The method may additionally or optionally include: wherein the corresponding candidate anchor feature is located at an initial three-dimensional position in the three-dimensional model and is extracted from frames of image data received at the initial time; determining that the corresponding candidate anchor feature does not exist in subsequent frames of image data received after the initial time; and maintaining the corresponding candidate anchor feature at the initial three-dimensional position in the three-dimensional model until a predetermined time period after the initial time expires.

[0124] On the other hand, a computing device is provided for manipulating a virtual object displayed via a display device. The computing device includes: a processor; and a memory holding instructions executable by the processor to perform the following operations: receiving image data of a real-world physical environment including a plurality of physical features; using the image data to generate a three-dimensional model of at least a portion of the real-world physical environment; extracting a plurality of candidate anchor features from the image data, each candidate anchor feature corresponding to one in the real-world physical environment; receiving user manipulation input to manipulate a virtual object displayed within the real-world physical environment, wherein the virtual object includes virtual anchor features; identifying at least one correspondence between the virtual anchor features of the virtual object and corresponding candidate anchor features from the plurality of candidate anchor features based on the manipulation of the virtual object; and displaying an indication of the corresponding candidate anchor feature at a corresponding physical feature within the real-world physical environment via the display device based on the identification of at least one correspondence. The computing device may additionally or alternatively include instructions executable by the processor to: align the virtual anchor features of the virtual object with corresponding candidate anchor features of the corresponding physical feature based on received user selection input; and display the virtual object whose virtual anchor features are aligned with the corresponding candidate anchor features via the display device. The computing device may additionally or alternatively include, wherein the corresponding physical feature is found on a physical object in a real-world physical environment, and instructions executable by a processor to display the corresponding movement of a virtual object via a display device as the physical object moves in the real-world physical environment, such that its virtual anchor point feature remains aligned with the corresponding candidate anchor point feature of the corresponding physical feature on the physical object. The computing device may additionally or alternatively include, wherein the corresponding physical feature is found on a physical object in a real-world physical environment, and instructions executable by a processor to display the virtual object at its virtual object location in the real-world physical environment as the physical object moves within the real-world physical environment, and instructions executable by a processor to maintain the display of the virtual object at its virtual object location. The computing device may additionally or alternatively include, wherein the corresponding physical feature is located in an invisible location in the real-world physical environment outside the field of view of the display device, and instructions executable by a processor to display an indication extending from the invisible location into the field of view of the display device via the display device. The computing device may additionally or alternatively include instructions executable by a processor to output a directional notification indicating the invisible location of a corresponding physical feature when a virtual anchor feature of a virtual object is aligned with a corresponding candidate anchor feature of a corresponding physical feature located at an invisible location.The computing device may additionally or alternatively include, wherein identifying at least one correspondence further includes: identifying at least one correspondence from a plurality of filtered candidate anchor features generated based on one or more of the following: (1) the distance between the display device and each of the candidate anchor features, (2) the distance between the virtual object and each of the candidate anchor features, (3) whether each of the candidate anchor features is within the field of view of the display device, (4) the frequency of previous use of each of the candidate anchor features, and (5) user-filtered input.

[0125] On the other hand, a head-mounted display device is provided, comprising: at least a partially transparent display; a processor; and a memory storing instructions executable by the processor to perform the following operations: displaying a virtual object via the at least partially transparent display; receiving image data of a real-world physical environment including multiple physical features; generating a three-dimensional model of at least a portion of the real-world physical environment using the image data; extracting multiple candidate anchor features from the image data, each candidate anchor feature corresponding to a physical feature in the real-world physical environment; receiving user manipulation input to manipulate a virtual object displayed in the real-world physical environment, wherein the virtual object includes virtual anchor features; identifying at least one correspondence between the virtual anchor features of the virtual object and corresponding candidate anchor features from the multiple candidate anchor features based on the manipulation of the virtual object; displaying an indication of the corresponding candidate anchor feature at the corresponding physical feature in the real-world physical environment via the at least partially transparent display based on identifying at least one correspondence; aligning the virtual anchor features of the virtual object with the corresponding candidate anchor features of the corresponding physical feature based on receiving user selection input; and displaying the virtual object whose virtual anchor features are aligned with the corresponding candidate anchor features.

[0126] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many variations are possible. The particular routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions shown and / or described may be performed in the order shown and / or described, in a different order, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0127] The subject matter of this disclosure includes all novel and inconspicuous combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as other features, functions, actions and / or properties, and any and all equivalents thereof.

Claims

1. A method for manipulating a virtual object displayed via a display device, the method comprising: Receive image data of a real-world physical environment that includes multiple physical features; Using the image data, a three-dimensional model of at least a portion of the real-world physical environment is generated; Multiple candidate anchor features are extracted from the image data, and each candidate anchor feature corresponds to one of the physical features in the real-world physical environment; Receive user input that manipulates the virtual object displayed within the real-world physical environment, wherein the virtual object includes virtual anchor point features; Based on the manipulation of the virtual object, at least one correspondence is identified between the virtual anchor point feature of the virtual object and the corresponding candidate anchor point feature from the plurality of candidate anchor point features, wherein the corresponding candidate anchor point feature is located at an initial three-dimensional position in the three-dimensional model and is extracted from the frame of the image data received at the initial time; Based on identifying the at least one correspondence, an indication of the corresponding candidate anchor point feature is displayed at the corresponding physical feature in the real-world physical environment via the display device; It is determined that the corresponding candidate anchor point feature does not exist in subsequent frames of the image data received after the initial time; and The corresponding candidate anchor point feature is maintained at the initial 3D position in the 3D model until a predetermined time period after the initial time expires.

2. The method according to claim 1, further comprising: Based on the received user selection input, the virtual anchor point feature of the virtual object is aligned with the corresponding candidate anchor point feature of the corresponding physical feature; as well as The virtual object whose virtual anchor point features are aligned with the corresponding candidate anchor point features is displayed.

3. The method according to claim 2, wherein the corresponding physical feature is found on a physical object in the real-world physical environment, the method further comprising: When the physical object moves in the real-world physical environment, the corresponding movement of the virtual object is displayed so that the virtual anchor point feature of the virtual object remains aligned with the corresponding candidate anchor point feature of the corresponding physical feature on the physical object.

4. The method of claim 2, wherein the corresponding physical feature is found on a physical object in the real-world physical environment, and the virtual object is displayed at the location of a virtual object in the real-world physical environment, the method further comprising: When the physical object moves in the real-world physical environment, the virtual object is maintained at the virtual object's location.

5. The method according to claim 1, wherein the plurality of physical features include three-dimensional linear edges and three-dimensional planar surfaces.

6. The method of claim 1, wherein the corresponding physical feature is located at an invisible location in the real-world physical environment outside the field of view of the display device, the method further comprising displaying the indication extending from the invisible location into the field of view of the display device.

7. The method of claim 6, further comprising: When the virtual anchor point feature of the virtual object aligns with the corresponding candidate anchor point feature of the corresponding physical feature located at the invisible position, a directional notification indicating the invisible position of the corresponding physical feature is output.

8. The method of claim 1, wherein the indication of the corresponding candidate anchor point feature is shown to extend beyond the corresponding physical feature.

9. The method according to claim 1, further comprising: Filtered candidate anchor features are generated by filtering the candidate anchor features based on one or more of the following: (1) the distance between the display device and each of the candidate anchor features, (2) the distance between the virtual object and each of the candidate anchor features, (3) whether each of the candidate anchor features is within the field of view of the display device, (4) the frequency of previous use of each of the candidate anchor features, and (5) user filtered input.

10. The method of claim 9, wherein identifying the at least one correspondence further comprises identifying the at least one correspondence from the filtered plurality of candidate anchor features.

11. The method of claim 1, further comprising receiving user pre-selection input for selected physical features from the plurality of physical features, wherein the selected physical features include the corresponding physical features of the corresponding candidate anchor features, and wherein identifying the at least one correspondence is also based on the user pre-selection input for the selected physical features.

12. A computing device for manipulating virtual objects displayed via a display device, the computing device comprising: processor; as well as The memory stores instructions that are executed by the processor to: Receive image data of a real-world physical environment that includes multiple physical features; Using the image data, a three-dimensional model of at least a portion of the real-world physical environment is generated; Multiple candidate anchor point features are extracted from the image data, each candidate anchor point feature corresponding to one of the physical features in the real-world physical environment, wherein at least one of the corresponding physical features is located in the real-world physical environment at an invisible location outside the field of view of the display device; Receive user input that manipulates the virtual object displayed within the real-world physical environment, wherein the virtual object includes virtual anchor point features; Based on the manipulation of the virtual object, identify at least one correspondence between the virtual anchor point feature of the virtual object and the corresponding candidate anchor point feature from the plurality of candidate anchor point features, wherein the corresponding candidate anchor point feature corresponds to the at least one corresponding physical feature located at the invisible position; Based on identifying the at least one correspondence, via the display device, an indication of the corresponding candidate anchor point feature is displayed at the corresponding physical feature in the real-world physical environment, wherein the indication extends from the invisible location into the field of view of the display device; as well as When the virtual anchor point feature of the virtual object aligns with the corresponding candidate anchor point feature of the corresponding physical feature located at the invisible position, a directional notification indicating the invisible position of the corresponding physical feature is output.

13. The computing device of claim 12, wherein the instructions are executed by the processor to: Based on the received user selection input, the virtual anchor point feature of the virtual object is aligned with the corresponding candidate anchor point feature of the corresponding physical feature; and The virtual object is displayed via the display device, wherein the virtual anchor point feature of the virtual object is aligned with the corresponding candidate anchor point feature.

14. The computing device of claim 13, wherein the corresponding physical feature is found on a physical object in the real-world physical environment, and the instructions are executed by the processor to: display the corresponding movement of the virtual object via the display device as the physical object moves in the real-world physical environment, such that the virtual anchor point feature of the virtual object remains aligned with the corresponding candidate anchor point feature of the corresponding physical feature on the physical object.

15. The computing device of claim 13, wherein the corresponding physical feature is found on a physical object in the real-world physical environment, and the virtual object is displayed via the display device at a virtual object location in the real-world physical environment, and the instruction is executed by the processor to: maintain the display of the virtual object at the virtual object location as the physical object moves in the real-world physical environment.

16. The computing device of claim 12, wherein identifying the at least one correspondence further comprises identifying the at least one correspondence from a plurality of filtered candidate anchor features generated based on one or more of the following: (1) the distance between the display device and each of the candidate anchor features, (2) the distance between the virtual object and each of the candidate anchor features, (3) whether each of the candidate anchor features is within the field of view of the display device, (4) the frequency of previous use of each of the candidate anchor features, and (5) user filtered input.

17. A head-mounted display device, comprising: At least a partially transparent display; processor; as well as A memory that stores instructions, which are executed by the processor, to: Virtual objects are displayed via the at least partially transparent display; Receive image data of a real-world physical environment that includes multiple physical features; Using the image data, a three-dimensional model of at least a portion of the real-world physical environment is generated; Multiple candidate anchor features are extracted from the image data, and each candidate anchor feature corresponds to one of the physical features in the real-world physical environment; Receive user input that manipulates the virtual object displayed within the real-world physical environment, wherein the virtual object includes virtual anchor point features; Based on the manipulation of the virtual object, at least one correspondence is identified between the virtual anchor point feature of the virtual object and the corresponding candidate anchor point feature from the plurality of candidate anchor point features, wherein the corresponding candidate anchor point feature is located at an initial three-dimensional position in the three-dimensional model and is extracted from the frame of the image data received at the initial time. Based on the at least one correspondence between the virtual anchor point feature and the corresponding candidate anchor point feature, an indication of the corresponding candidate anchor point feature is displayed at the corresponding physical feature of the corresponding candidate anchor point feature in the real-world physical environment via the at least partially transparent display; It is determined that the corresponding candidate anchor point feature does not exist in subsequent frames of the image data received after the initial time; and The corresponding candidate anchor point feature is maintained at the initial 3D position in the 3D model until a predetermined time period after the initial time expires.

Images