Method and device for adaptive augmented reality anchor point generation
Through the adaptive AR placement user interface to detect surfaces and obstacles, adjust the AR anchor generation area and capture instructions, it solves the problem that AR objects are difficult to achieve in real-world scenarios, ensuring that AR objects are displayed in appropriate locations and orientations. , realizes the durability and sharing of AR experience.
Patent Information
- Application Number
- CN202080007798.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-07-22
AI Technical Summary
The proper position and orientation of placing augmented reality (AR) objects in real-world scenarios is difficult to achieve, especially in the presence of physical obstacles, which leads to the inability to generate and display in the desired way.
Adaptive AR placement user interface (UI) is used to generate AR anchor points. By detecting surfaces and obstacles in real-world areas, adjusting AR anchor generation areas and capture instructions to adapt to the physical environment and ensure that AR objects are in the appropriate position. and directional display.
The ability to effectively generate and display AR objects in the presence of obstacles is realized, ensuring that AR objects stay in place and orientation, thereby maintaining the illusion of virtual objects and allowing users to share and reproduce the AR experience.
Smart Images

Figure CN114341773B_ABST
Abstract
Description
Background Art
[0001] Placing an augmented reality (AR) object in the proper context within an image of a real-world scene viewed through a user's mobile device can be complex. Specifically, placing an AR object in the proper position and / or orientation within a display can be difficult to achieve. Summary of the invention
[0002] This document describes methods and apparatus for generating AR anchors.
[0003] In one aspect, a method includes: detecting a surface within a real world area, the surface being captured by a user via an image sensor in a mobile device; receiving an AR generation instruction to generate an augmented reality (AR) anchor point that intersects the surface within the real world area, the AR anchor point being at a target location for displaying an AR object; and defining a capture instruction in response to the AR generation instruction and based on the intersection. The AR anchor point that intersects the surface may include an AR generation area associated with the AR anchor point that intersects the surface. The AR generation area may include a capture path for capturing the AR anchor point. The defined capture instruction may define the AR generation area. In addition, defining the capture instruction may include modifying the capture instruction according to a default capture instruction. In addition, the capture instruction may include a capture arc. The capture arc may be less than 360 degrees. In addition, the capture instruction may be displayed within an adaptive AR placement user interface. The surface may be a first surface, and the method may further include: detecting a second surface within the real world area, defining the capture instruction based on the first surface and the second surface. In addition, the capture instruction may include a capture arc less than 180 degrees based on the first surface and the second surface.
[0004] In another aspect, a method includes: detecting a surface within a real-world area, the surface captured by a user via an image sensor in a mobile device; receiving an AR generation instruction to generate an AR anchor associated with an AR generation area that intersects the surface within the real-world area; and modifying a capture instruction in response to the AR generation instruction and based on the intersection of the AR generation area with the surface. Modifying the capture instruction may define a capture arc. In addition, the capture instruction may be displayed within an adaptive AR placement user interface. In addition, the capture instruction may include a capture progress bar.
[0005] In another aspect, a method includes: detecting a surface within a real world area, the surface associated with an obstacle captured by a user via an image sensor in a mobile device; receiving an AR generation instruction to generate an AR anchor at a target location; and modifying an AR placement user interface based on an AR generation area around the target location that intersects the surface. The method may further include limiting the capture instructions displayed via the AR placement user interface based on the AR generation area around the target location. In addition, the method may also include limiting the AR generation area so that the AR generation area does not intersect the surface.
[0006] The above-described aspects of the method according to the invention can of course also be used as aspects of a corresponding device.
[0007] In one aspect, an apparatus includes: a sensor system configured to capture at least a portion of a real-world area using a mobile device; a surface recognition engine configured to detect a surface within the real-world area; an anchor point generation engine configured to receive AR generation instructions to generate an AR anchor point at a target location within a threshold distance of the surface within the real-world area; and an adaptive AR placement user interface configured to limit capture instructions in response to the AR generation instructions and based on the target location being within the threshold distance. The capture instructions can be modified based on a default capture instruction that limits scanning completely around the target location. In addition, the capture instructions include a capture arc of less than 360 degrees.
[0008] In another aspect, an apparatus includes: a sensor system configured to capture at least a portion of a real-world area using a mobile device; a surface recognition engine configured to detect a surface within the real-world area via an image sensor in the mobile device; an anchor point generation engine configured to receive AR generation instructions to generate an AR anchor point that intersects the surface within the real-world area, the AR anchor point being at a target location for displaying an AR object; and an adaptive AR placement user interface configured to respond to the AR generation instructions and to limit a capture instruction based on the intersection. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 A map of the physical area in which a user views an augmented reality (AR) object associated with an AR anchor.
[0010] Figure 2 is a block diagram illustrating a system configured to implement the concepts described herein.
[0011] Figures 3A to 3C A user interface view showing an adaptive AR placement user interface that implements at least some portions of the above process based on the presence or absence of various obstacles.
[0012] Figures 4A to 4C Another example of an adaptive AR placement UI is shown.
[0013] Figures 5A to 5C Another example of instructions displayed within an adaptive AR placement UI is shown.
[0014] Figure 5D and 5E Additional examples of instructions displayed within an adaptive AR placement UI are shown.
[0015] Figure 6 Distance-based AR anchor point generation is shown.
[0016] Figure 7 An example of AR anchor positioning is shown.
[0017] Fig. 8A and 8B The real world scene without and with AR objects is shown respectively.
[0018] Figures 9 to 12 A method of generating an AR anchor as described herein is shown.
[0019] Fig.13 Examples of a general purpose computer device and a general purpose mobile computer device are shown. DETAILED DESCRIPTION
[0020] Placing an augmented reality (AR) object in an appropriate location and / or orientation within an image of a real-world scene viewed through a user's mobile device can be difficult to achieve. When physical obstacles make it difficult to create an AR anchor within the physical location of the AR object, the AR object may not be associated with the physical location (e.g., anchor location). For example, a wall or other object may block the scanning of an area for the AR anchor (e.g., a 360-degree scanning arc around the periphery of the physical area) or block the scanning of an area as an AR anchor so that the AR object can be associated with the AR anchor. Therefore, when there are obstacles, the AR anchor may not be created in a desired manner so that the user who places the AR object or others who intend to interact with the AR object can view the AR object later. In some implementations, the AR anchor may include a mapping of a physical area around the target location of the AR object and / or a real-world area coordinate system location (e.g., a global positioning system (GPS) location) for positioning. The target location can be an approximate or target area of the AR anchor and the place of the AR object associated with the AR anchor.
[0021] AR anchors can ensure that AR objects (associated with the AR anchors) stay in the same position and orientation in space, helping to maintain the illusion of virtual objects placed in the real world. AR anchors allow devices to recognize and resolve space, so users can share AR experiences with other users simultaneously and / or return to the same AR experience later (e.g., providing persistence). AR anchors (and associated AR objects) can be created by one user (or device) and resolved at a later time by the same user (or device) or at a later time by another user. In some implementations, persistent AR anchors that can be saved for use by one or more users can be a type of AR cloud anchor.
[0022] In order to achieve accurate placement of AR objects (also referred to as points of interest (POIs)) at AR anchors, an adaptive AR placement user interface (UI) may be used to generate (e.g., create) AR anchors. The adaptive AR placement UI may adapt the AR anchor generation algorithm to obstacles within the physical environment so that the AR anchor may be generated in a desired manner. For example, the adaptive AR placement UI may trigger (e.g., allow) scanning of a portion of a physical area (e.g., an area less than 360°) as an AR anchor. When the AR anchor is generated, augmented reality may be used to display the AR object in an appropriate context at the AR anchor in the real world. In some implementations, the AR object appears to be anchored to a physical element when rendered. In some implementations, a target area is detected, and a range (e.g., area and / or angle range) (e.g., a progress ring) for AR anchor generation may be adapted to the user's environment to facilitate appropriate AR anchor generation.
[0023] In some implementations, for example, GPS and Wi-Fi can be used at least in part to locate the device to the AR anchor point. In some implementations, a magnetometer sensor can be used at least in part to orient the device relative to the AR anchor point (e.g., a magnetometer sensor can be relatively inaccurate and / or can be impaired by local magnetic fields).
[0024] Although the concepts described herein are generally discussed with respect to AR anchor generation, the concepts described herein can be applied to any type of scanning. For example, the adaptive UI as described herein can be applied to any type of three-dimensional (3D) scanning, such as 3D volume scanning. Specifically, the adaptive UI described herein can be used to guide a user during 3D scanning of an object using a mobile device.
[0025] Figure 11 is a diagram of a user 100 in a real-world area 10 (e.g., a real-world location) viewing an AR object P1 through a mobile device 110. The position (e.g., position and / or orientation, and / or distance and orientation) of the user 100 is located relative to the real-world area 10, and the AR object P1 is located in the real-world area 10, such as within an AR anchor point A1 (in Figure 1 The dotted circle in the figure has a fixed position.
[0026] The AR object P1 is appropriately displayed within the mobile device 110 (e.g., on its display screen) using a combination of the positioning of the user 100's mobile device 110 to the real-world area 10 (which may be referred to as the positioning of the user 100 and / or the positioning of the mobile device 110) and the fixed position of the AR object P1 within the real-world area 10 via the AR anchor point A1. When the mobile device 110 is positioned to the AR anchor point A1, the AR object P1 may also be included at the AR anchor point A1 of the real-world area 10 (at a fixed position and orientation (e.g., X, Y, and Z coordinate orientation)).
[0027] For example, in some implementations, a representation of a real-world scene from the real-world area 10 can be captured by the user 100 using a camera of the mobile device 110. The real-world scene can be a portion of the real-world area 10 captured by a camera (e.g., a camera of the mobile device 110). The position (and / or orientation) of the mobile device 110 can be associated with the AR anchor A1 based on a comparison (e.g., matching of features) of the representation of the real-world scene with a portion and / or position (e.g., GPS position) of the AR anchor A1. In some implementations, positioning can include determining the position and orientation of the mobile device 110 relative to the AR anchor A1. In some implementations, the position and orientation can include a distance from the AR anchor A1 and a direction in which the mobile device 110 is facing relative to the AR anchor A1. Because the AR anchor A1 has a fixed position relative to the real-world area 10, the position and orientation of the mobile device 110 relative to the real-world area 10 can be determined. Thus, the position and orientation of the mobile device 110 relative to the AR object P1 can be determined with the help of the AR object P1 having a fixed position and orientation within the AR anchor point A1 of the real-world area 10, and then the AR object P1 can be displayed to the user 100 at an appropriate position and orientation within the mobile device 110. Changes in the position and orientation of the mobile device 110 can be determined by sensors (e.g., an inertial measurement unit (IMU), a camera, etc.) and can be used to update the position and / or orientation of the AR object P1 (and / or other AR objects).
[0028] When the user 100 creates the AR anchor point A1, the user 100 may be prompted to move to the target location (in Figure 1The AR anchor point A1 (and / or the AR object P1) is scanned within an area (e.g., a real-world physical area) within a real-world area 10 (represented as point T in FIG. 1 ). The target location T may be an approximate location for placing the AR object P1 associated with the AR anchor point A1. Thus, the AR object P1 and / or the AR anchor point A1 may be associated with the target location T or approximately centered about the target location T.
[0029] The scanning of the AR anchor point A1 can be performed within (e.g., performed around) the AR anchor point generation area C1. In some implementations, the scanning can be performed around the periphery of the AR anchor point generation area C1. The user 100 can move around the periphery of the AR anchor point generation area C1 (as shown by the arrows) while aiming the camera of the mobile device 110 at the target position T of the AR object P1. The AR anchor point A1 can be an image and / or representation associated with the target position T (e.g., a point and / or area) within the real-world area 10, which can be used to later identify a location for presenting the AR object P1.
[0030] Although referred to as an AR anchor generation area, the AR anchor generation area may be or may correspond to a volume, a location, a set of locations, a pattern, a path (e.g., an arc path), etc. In some implementations, the AR anchor generation area may correspond to an area that is scanned to define an AR anchor.
[0031] Because the target AR anchor point generation area (including both areas C1 and C2) intersects (e.g., is adjacent to, includes) a surface 11 (e.g., an obstacle, an object) within the real-world area 10, the generation of the AR anchor point A1 is adapted (e.g., modified) in this example to accommodate the surface 11. The target AR anchor point generation area including both C1 and C2 may be an area calculated for the desired capture of the AR anchor point A1 and placement of the AR object P1 (at the target position T). The target AR anchor point generation area may be an area centered at the target position T for placement of the AR object P1. The target AR anchor point generation area may be a default circular AR anchor point generation area around the target position T. In some implementations, the AR anchor point generation area C1 may be derived from the target AR anchor point generation area based on the presence of the surface 11.
[0032] In some implementations, adapting the generation of the AR anchor point A1 may include adapting instructions to generate the AR anchor point A1 around the AR anchor point generation area C1. In some implementations, the instructions to generate the AR anchor point A1 may be presented on a user interface of the mobile device 110. In some implementations, the AR anchor point generation area C1 may correspond to an area scanned as part of the AR anchor point A1. In some implementations, if the AR anchor point generation area C1 (if a complete circle (having both areas C1 and C2)) will intersect the surface 11, the AR generation may be adapted. Without adapting the AR anchor point generation area C1 to exclude an arc (associated with area C2) passing through the surface 11, the AR generation area (e.g., a default AT generation area) may include area C2, which will pass through the surface 11 and will be an impossible area (e.g., path) for the user 100 for AR anchor point generation.
[0033] As an example, the AR anchor point generation area C1 may be defined such that the scan of the AR anchor point A1 is along Figure 1 The semicircular shape (or arc) shown is performed instead of the entirety of the target location T around the AR anchor point A1 (e.g., 360 degrees, areas C1 and C2). Due to the surface 11, scanning around the entire target location T of the AR anchor point A1 would be difficult or impossible. In some implementations, the user interface for guiding the user 100 to generate the AR anchor point A1 can be adapted based on the surface 11. In some implementations, the generation of the AR anchor point A1 can be adapted based on the presence of multiple surfaces (e.g., obstacles) detected within the real-world area 10. In some implementations, the AR anchor point generation area C1 can be defined so that the AR anchor point generation area C1 does not intersect with the surface 11 or other surfaces (e.g., obstacles).
[0034] In some implementations, the scanning area associated with AR anchor generation can be reduced so that a more complete scan of the AR anchor can be completed. For example, in some implementations, the radius of the AR anchor generation area C1 can be reduced (in response to detecting the surface 11) so that a more complete scanning arc (e.g., greater than 180 degrees, 270 degrees) around the AR anchor A1 (and the AR object P1) can be achieved to create the AR anchor A1.
[0035] In some implementations, the scanning area associated with AR anchor generation can be changed based on the size of the target AR object. For example, in some implementations, although an obstacle is detected, the radius of the AR anchor generation area can be reduced so that when the AR object is relatively small, a more complete scanning arc (e.g., greater than 180 degrees, 270 degrees) around the AR anchor (and the AR object) can be achieved. In this case, the AR object can be small enough that a relatively complete scan for placing the AR object at the AR anchor can be achieved even if the target position of the AR object is near an obstacle. As another example, if the AR object is relatively large and near an obstacle, the radius of the AR anchor generation area may be relatively large and the scanning arc may need to be reduced accordingly.
[0036] In some implementations, AR anchor generation may be adapted based on at least a portion of the AR anchor A1 having at least a portion of an area that intersects (e.g., is adjacent to (e.g., within a threshold distance), including) at least a portion of a surface, such as the surface 11. In some implementations, AR anchor generation may be adapted based on at least a portion (e.g., an area) of the AR anchor A1 scanned during AR anchor generation intersecting (e.g., is adjacent to (e.g., within a threshold distance), including) at least a portion of a surface, such as the surface 11. In some implementations, AR anchor generation may be adapted based on a target position T of an AR object that intersects at least a portion of a surface, such as the surface 11.
[0037] In some implementations, AR anchor point generation may be adapted based on satisfying a condition associated with the surface 11. For example, in some implementations, AR anchor point generation may be adapted based on a size (e.g., volume, surface area) of the surface 11, and in some implementations, AR anchor point generation may be adapted based on exceeding a threshold size of the surface 11.
[0038] As a specific example, if the surface 11 is large enough (exceeding a threshold size) to disrupt AR anchor generation, AR anchor generation may be adapted (e.g., instructions or guidelines for triggering AR anchor generation). As a specific example, if the surface 11 has a large enough size that the AR generation area C1 cannot be a full circle (e.g., the full circle includes areas C1 and C2). If the surface 11 is short (e.g., such that the user 100 can climb over the surface during AR anchor generation), AR anchor generation may not be adapted.
[0039] In some implementations, AR anchor point generation may not be adapted based on not satisfying a condition associated with surface 11. For example, in some implementations, AR anchor point generation may not be adapted based on the size (e.g., volume, surface area) of surface 11, and in some implementations, AR anchor point generation may not be adapted based on the size of surface 11 falling below a threshold size. As another example, if the obstacle is a protruding area under which the user can move, AR generation may not be adapted.
[0040] As a specific example, if the surface 11 is Figure 1 If the surface 11 is shorter than shown, the user 100 may crawl over the surface 11 during AR anchor point generation. In this case, AR anchor point generation may not be adapted, and a full 360-degree scan (or a larger scan arc) of the target location T of the AR anchor point A1 (e.g., around the periphery or area around the target location T) may be scanned to generate the AR anchor point A1 and place the AR object P1.
[0041] In some implementations, the position may include a position in X, Y, Z coordinates, and the orientation may include a directional orientation (e.g., a direction or angle that an object or user is facing, yaw, pitch, and roll). Thus, a user (e.g., user 100) and / or an AR object (e.g., AR object P1) may be at a specific X, Y, Z position and facing a specific direction as the orientation at the X, Y, Z position.
[0042] Figure 2 is a diagram showing a system configured to implement the concepts described herein (e.g., Figure 1 2. A block diagram of a system 200 is shown in FIG. 200 . The system 200 includes a mobile device 110 and an AR server 252. Figure 2 Details of the mobile device 110 and the AR server 252 are shown. Using the system 200, one or more AR objects can be displayed within the display device 208 of the mobile device 110 using a combination of positioning the mobile device 110 to an AR anchor within the real world area and the fixed position and orientation of the AR objects within the real world area. Figure 1 The operation of system 200 is described in the context of FIG. 2 and other figures.
[0043] The mobile device 110 may include a processor component 204, a communication module 206, a sensor system 210, and a memory 220. The sensor system 210 may include various sensors, such as a camera component 212, an inertial motion unit (IMU) 214, and a global positioning system (GPS) receiver 216. Implementations of the sensor system 210 may also include other sensors, including, for example, light sensors, audio sensors, image sensors, distance and / or proximity sensors, contact sensors such as capacitive sensors, timers, and / or other sensors and / or different combinations of sensors. The mobile device 110 includes a device positioning system 242 that may utilize one or more portions of the sensor system 210.
[0044] The mobile device 110 also includes a display device 208 and a memory 220. Applications 222 and other applications 240 are stored in the memory 220 and can be accessed therefrom. Applications 222 include an AR anchor positioning engine 224, an AR object retrieval engine 226, an AR rendering engine 228, and an anchor generation engine 227. The anchor generation engine 227 includes a surface recognition engine 225, an AR generation area engine 231, and an adaptive AR UI engine 230. In some implementations, the mobile device 110 is a mobile device such as a smartphone, a tablet computer, a head mounted display device (HMD), AR enabled glasses, etc.
[0045] The system shows details of an AR server 252, which includes a memory 260, a processor component 254, and a communication module 256. The memory 260 is configured to store an AR anchor point A (which may include Figure 1 AR anchor point A1) and AR object P (and may include Figure 1 AR object P1).
[0046] Although the processing blocks shown in the AR server 252 and the mobile device 110 are illustrated as being included in specific devices, the processing blocks (and processing associated therewith) may be included in different devices, divided between devices, etc. For example, at least a portion of the surface recognition engine 225 may be included in the AR server 252.
[0047] The anchor point generation engine 227 is configured to generate an anchor point for the user (e.g., Figure 1 The user 100 shown in FIG. 1 is creating an AR object (eg, Figure 1The user 100 may recognize that the AR object P1 is to be placed at the target location T, and the anchor generation engine 227 may then be used to place the AR object P1 by generating the AR anchor A1 associated with the AR object P1. Thus, the anchor generation engine 227 may be used to place the AR anchor A1 at the target location T within the real-world area 10.
[0048] In the process of creating the AR anchor point A1, the sensor system 210 may be configured to capture at least a portion of the real world area 10 using the mobile device 110. Specifically, the sensor system 210 may be triggered by the anchor point generation engine 227 to capture at least a portion of the real world area 10.
[0049] In order to generate the AR anchor point A1, the AR generation area engine 231 may define an AR generation area C1 around the target position T. The surface recognition engine 225 may be configured to identify a surface 11 (and / or other surfaces that may be associated with one or more objects or obstacles) that intersects (e.g., is adjacent to) the AR generation area C1 defined by the AR generation area engine 231. The AR generation area C1 may be defined by the AR generation area engine 231, and the user interface for instructing the generation of the AR anchor point A1 in the AR generation area C1 may be adapted to the defined AR generation area C1. Specifically, due to the presence of the surface 11, the scanning area corresponding to the AR generation area C1 is defined by the AR generation area engine 231 as approximately a semicircle (e.g., a semicircular arc), and the instructions (e.g., a guide) generated by the adaptive AR UI engine 230 within the adaptive AR placement UI (not shown) may be adapted to match the semicircular shape of the AR generation area C1.
[0050] In other words, the anchor point generation engine 227 may be configured to modify AR anchor point generation in response to an obstacle detected by the surface recognition engine 225. The AR user interface may be adapted by the adaptive AR placement UI 230 based on the obstacle to guide the user 100 to generate the AR anchor point A1. In some implementations, the adaptive AR placement UI 230 may adapt the AR user interface based on the obstacle to guide the user 100 to generate the AR anchor point A1 based on the AR generation area determined by the AR generation area engine 231 based on the obstacle.
[0051] Figures 3A to 3C A user interface view showing an adaptive AR placement UI that implements at least some portions of the above process based on the presence or absence of various obstacles. Figure 3A FIG. 3 shows a user interface view of the adaptive AR placement UI 310 that implements the above process. Figure 3AAs shown, the user is instructed to scan an area (e.g., an arc) of approximately 180 degrees (indicated by dashed arrow C2) around the AR object 320 to create an AR anchor point associated with the target location T2 based on the presence of the obstacle 330 (e.g., a vertical wall).
[0052] Figure 3B FIG. 3 shows a user interface view of the adaptive AR placement UI 311 that implements the above process. Figure 3B As shown, the user is instructed to scan an area (eg, an arc) less than 180 degrees around the AR object 320 to create an AR anchor point associated with the target location T3 based on the presence of two obstacles 331 , 332 (eg, vertical walls).
[0053] Figure 3C FIG. 3 shows a user interface view of the adaptive AR placement UI 312 that implements the above process. Figure 3C As shown, the user is instructed to scan the area around the entirety (e.g., 360 degrees) of the AR object 320 to create an AR anchor associated with the target location T4 based on the absence of obstacles (e.g., no recognition surface that interferes with the generation of the AR anchor). In some implementations, the scan around the entirety of the AR object can be referred to as a default anchor generation mode or scan.
[0054] Figures 4A to 4C Another example of an adaptive AR placement UI 410 is shown. Figure 4A As shown, the user can be guided by arrows to scan around the AR object 420. Elements 442 (e.g., shown as small vertically oriented oval elements) (which may also be referred to as parts, segments, or sections) included in the progress bar 441 are colored with various colors as the scanning progresses. Figure 4B As shown, as the scanning progresses, more elements 443 of the progress bar 441 are colored with various colors. Figure 4C As shown, the scanning is completed, and all elements of the progress bar 441 are colored with a color indicating that the scanning is completed. After the scanning is completed, the AR anchor point of the AR object 420 can be stored.
[0055] In some implementations, the colors of the elements of the progress bar 441 can indicate different instructions or information. White elements can indicate (or correspond to) areas that have not yet been scanned. Orange and yellow elements can indicate (or correspond to) areas that have been scanned at low quality and medium quality, respectively. Green elements can indicate (or correspond to) areas that have been scanned at high quality.
[0056] Return to reference Figure 2And as described above, in some implementations, adapting the generation of the AR anchor point may include adapting instructions (e.g., guidelines) within the adaptive AR placement UI to generate the AR anchor point. In some implementations, the instructions to generate the AR anchor point A1 may be presented by the adaptive AR UI engine 230 within the adaptive AR placement UI of the mobile device 110. Figures 5A to 5C Several such examples are shown in .
[0057] Figure 5A A progress ring in the adaptive AR placement UI 510 during AR anchor point generation is shown, which sets a default progress ring with a range of 360 degrees based on the absence of obstacles. Figure 5B A progress ring with a range of approximately 90 degrees in the adaptive AR placement UI 511 AR anchor point generation based on the presence of two obstacles is shown. Figure 5C A progress ring with a range of approximately 180 degrees in AR anchor point generation in adaptive AR placement UI 512 based on the presence of an obstacle (e.g., a vertical wall) is shown. The progress ring can guide the user to move in a specific direction.
[0058] Figure 5D and 5E 1 shows additional UI implementations that can be used in conjunction with the concepts described herein. In some implementations, rather than using a progress ring, the UI can include a Figure 5D The progress ball 581 shown and the Figure 5E Progress ball 582 is shown. Progress balls 581 and 582 target AR object 583. The size and / or shape of the progress ball can be adapted based on one or more obstacles. For example, a progress ball surrounding an object can be a default shape for scanning around an object. If there are obstacles that will intersect the progress ball, the progress ball can be truncated into a portion (e.g., element, segment, cross-section) of the progress ball (e.g., truncated into a hemisphere). In these implementations, the portions of the progress ball are labeled 581A-C ( Figure 5D ) and 582A-C( Figure 5E ). Other shapes and / or UIs may be implemented in conjunction with the concepts described herein, including square, oval, elliptical, etc. In some implementations, the progress UI may have an irregular shape that is non-uniform and / or asymmetrical.
[0059] like Figure 6As shown, in some implementations, AR anchor point generation can be adapted based on the distance X. In some implementations, the distance X can be the distance between the target position T and the surface 11. In some implementations, the distance X can be the distance between the target position T and a portion of the AR object P1 (e.g., the center of mass, the middle portion). For example, if the distance X between the target position T and the surface 11 satisfies a condition (e.g., within a threshold distance), AR anchor point generation can be adapted (e.g., the length (or degree) of the scanning arc can be limited). In some implementations, if the distance (e.g., the minimum distance) between the edge of the AR object P1 and the surface 11 satisfies a threshold condition (e.g., within a threshold distance), AR anchor point generation can be adapted.
[0060] The AR anchor points A (which may each be unique) may each be located at a fixed position (and / or orientation) within the coordinate space of the real-world area 10. In some implementations, at least each AR anchor point P has a position (not an orientation) within the real-world area 10.
[0061] As described above, AR anchor points A can be used to locate the user 100 (e.g., the user's mobile device 110) to the real world area 10. In some implementations, AR anchor points can be considered as AR activation markers. For example, AR anchor points A can be generated so that the user's mobile device 110 can be located at one or more of the AR anchor points A. For example, AR anchor points A can be images and / or representations associated with a location (e.g., point and / or area) of the real world area 10. In some implementations, AR anchor points A (like real world area 10) can be a collection of points (e.g., point clouds) representing features (e.g., edges, density, buildings, walls, signs, planes, objects, textures, etc.) at or near a location (e.g., point and / or area) within the real world area 10. In some implementations, AR anchor points A can be spherical images (e.g., color images) or panoramas associated with a location within the real world area 10. In some implementations, one or more AR anchor points A can be content items. In some implementations, AR anchor points A can be one or more features associated with a location within the real world area 10.
[0062] In some implementations, one or more AR anchor points A may be created by capturing features (e.g., an image or a set of images (e.g., a video), a panorama, a scan) while the user 100 (holding the mobile device 110) is physically standing at or moving around a point (e.g., a target location) and / or area within the real-world area 10. Creating the AR anchor point A may then be performed using the anchor point generation engine 227. The one or more captured features may be mapped to a location (e.g., a set of features associated with the location) within the real-world area 10 as Figure 1 The AR anchor point A1 is shown. This information may be stored in the AR server 252 .
[0063] In some implementations, one or more of the AR anchor points A within the real-world area 10 may include a uniquely identifiable marker (e.g., a physical marker) to be used as an AR activation marker. In some limitations, the marker may include text, QR, a custom-designed visual scan code, etc. In some implementations, the AR anchor point A may be a uniquely identifiable physical marker connected by, for example, a position and / or orientation within the real-world area 10. The physical markers in the real-world area 10 may be used to accurately calibrate the position and / or orientation of the mobile device 110.
[0064] Figure 2 The AR anchor positioning engine 224 in the can be configured to determine the position of the mobile device 110 based on a comparison (e.g., matching of features) of the representation of the real-world scene with a portion of the real-world area 10 of the real-world area. The comparison can include comparing features (e.g., edges, density, buildings, walls, signs, planes, objects, textures, etc.) captured by the mobile device 110 with features included in or represented within, for example, the real-world area 10. In some implementations, the comparison can include comparing a portion of an image captured by the mobile device 110 with a portion of an image associated with the real-world area 10.
[0065] The camera assembly 212 can be used to capture images or videos of a physical space, such as a real-world scene, from the real-world area 10 around the mobile device 110 (and the user 100) for positioning purposes. The camera assembly 212 can include one or more cameras. The camera assembly 212 can also include an infrared camera. In some implementations, a representation (e.g., an image) of a real-world scene from the real-world area 10 can be captured by the user 100 using the camera assembly 212 camera of the mobile device 110. The representation of the real-world scene can be a portion of the real-world area 10. In some implementations, features (e.g., one or more images) captured using the camera assembly 212 can be used to locate the mobile device 110 to one of the AR anchor points A stored in the memory 260 of the AR server 252.
[0066] Based on the comparison of the features, the AR anchor positioning engine 224 may be configured to determine the position and / or orientation of the mobile device 110 relative to one or more AR anchors A. The position (and / or orientation) of the mobile device 110 may be positioned relative to the position of one of the AR anchors by comparison of an image viewed by the mobile device 110, for example. Specifically, for example, an image captured by a camera of the mobile device 110 may be used to determine the position and orientation of the mobile device 110 relative to the AR anchor A.
[0067] exist Figure 7 , where the mobile device 110 captures a portion of a corner of a wall and a portion of a painting 702 (e.g., the interior of a building, the interior of a building on a particular floor (e.g., multiple floors of a building)). The capture area is shown as capture area 70. This capture area 70 can be used to locate the mobile device 110 to the AR anchor point E1, which was previously captured from a different angle (e.g., captured by another mobile device) and includes an overlap area 72 as shown by the dotted line. Specifically, features of the capture area 70 can be compared with features of the overlap area 72 associated with the AR anchor point E1 to locate the mobile device 110 to the AR anchor point E1.
[0068] In some implementations, the AR anchor positioning engine 224 can be configured to determine the position and / or orientation of the mobile device 110 relative to one of the multiple AR anchors A. In some implementations, when the user is at a specified location (or area) at a given time (or within a time window), only one of the AR anchors A is selected for positioning.
[0069] Even after positioning at one of the AR anchor points A, the precise position and orientation of the mobile device 110 within the physical real world may not be known. Only the relative position and orientation of the mobile device 110 relative to at least one AR anchor point A (and through the AR anchor point A within the real-world area 10) is known. Special capture of feature (e.g., image) information by the mobile device 110 is used to determine the relative position of the mobile device 110.
[0070] In some implementations, the AR anchor positioning engine 224 may also use images captured using the camera assembly 212 to determine the position and orientation of the mobile device 110 within a physical space, such as an interior space (e.g., the interior space of a building), based on a representation of the physical space received from the memory 260 or an external computing device. In some implementations, the representation of the physical space may include visual features of the physical space (e.g., features extracted from an image of the physical space). The representation may also include position determination data associated with those features that the visual positioning system may use to determine the position and / or location within the physical space based on one or more images of the physical space. The representation may also include a three-dimensional model of at least some structures within the physical space. In some implementations, the representation does not include a three-dimensional model of the physical space.
[0071] In some implementations, the AR anchor positioning engine 224 may use multiple sensory signals (from one or more sensor systems 210) to uniquely identify the AR anchor. In some implementations, these include, but are not limited to: image recognition and tracking, text recognition and tracking, AR tracking orientation points, GPS positioning, Wifi signals, QR codes, custom designed visual scan codes, etc.
[0072] refer to Figure 1 , changes in the position and orientation of the mobile device 110 relative to the AR anchor point A1 can be determined by sensors (e.g., inertial measurement unit (IMU), camera, etc.) and can be used to update the position and / or orientation of the AR object P1. For example, if the mobile device 110 moves to a different direction, the display of the AR object P1 can be modified accordingly within the display device 208 of the mobile device 110.
[0073] Return to reference Figure 2 , the AR object retrieval engine 226 may be configured to retrieve one or more AR objects P from the AR server 252. For example, the AR object retrieval engine 226 may retrieve the AR object P within the real-world area 10 based on the coordination of the coordinate spaces of the AR object P, the real-world area 10, and the AR anchor point A performed by the surface recognition engine 225.
[0074] The AR presentation engine 228 presents or causes one or more AR objects P to be presented on the mobile device 110. For example, the AR presentation engine 228 can cause the adaptive AR placement UI 230 to generate a user interface that includes information or content from one or more AR objects P to be displayed by the mobile device 110. In some implementations, the AR presentation engine 228 is triggered by the AR object retrieval engine 226 retrieving one or more AR objects P. The AR presentation engine 228 can then trigger the display device 208 to display content associated with the one or more AR objects P.
[0075] The adaptive AR placement UI 230 may be configured to generate a user interface. The adaptive AR placement UI 230 may also cause the mobile device 110 to display the generated user interface. The adaptive AR placement UI 230 may, for example, display information or content from one or more of the AR objects P. In some implementations, the adaptive AR placement UI 230 generates a user interface including a plurality of user-actuable controls, each of which is associated with one or more of the AR objects P. For example, a user may actuate one of the user-actuable controls (e.g., by touching the control on a touch screen, clicking the control using a mouse or another input device, or otherwise actuating the control).
[0076] exist Figure 8BAn example of an AR object 801 displayed within a real-world scene 800 is shown in FIG. The AR object 801 may be stored at the AR server 252 . Fig. 8A A real world scene 800 without an AR object 801 is shown in FIG.
[0077] Figures 9 to 12 The method of generating an AR anchor as described herein is shown. The flowchart element may consist of at least Figure 2 The anchor point generation engine 227 is shown to execute.
[0078] like Fig. 9 As shown, a method may include: capturing at least a portion of a real-world area C2 using a mobile device (block 910), and (e.g., by Figure 2 The method may include (e.g., in Figure 2 ) receives AR generation instructions to generate an AR anchor point at a target location within a threshold distance of a surface within a real world area (block 930), and (e.g., at an anchor point generation engine 227 shown in FIG. 1 ). Figure 2 The adaptive AR placement UI 230 shown in FIG. 2A ) generates an instruction in response to the AR and defines a capture instruction based on the target location being within a threshold distance (block 940 ).
[0079] like Fig.10 As shown, the method includes (for example, by Figure 2 The surface recognition engine 225 shown in FIG. 1 detects a surface in a real-world area, where the surface is captured by a user via an image sensor in a mobile device (block 1010), and (e.g., in Figure 2 The method further includes (e.g., at an anchor point generation engine 227 shown in FIG. 1 ) receiving AR generation instructions to generate an augmented reality (AR) anchor point that intersects a surface within a real world area (block 1020). The AR anchor point may be located at a location for displaying an AR object. Figure 2 The adaptive AR placement UI 230 shown in FIG. 2A ) generates an instruction in response to the AR and defines a capture instruction based on the intersection (block 1030 ).
[0080] In some implementations, an AR anchor point that intersects a surface includes an AR anchor point associated with the surface (e.g., Figure 2 The AR generation area C1 includes a capture path to capture the AR anchor point. The defined capture instructions define the AR generation area. In some implementations, defining the capture instructions includes modifying the capture instructions from the default capture instructions. The capture instructions may include a capture arc. The capture arc may be less than 360 degrees. In some implementations, the capture instructions are displayed in the adaptive AR placement user interface.
[0081] like Fig.11 As shown, a method may include (for example, by Figure 2 The surface recognition engine 225 shown in FIG. 1 detects a surface in a real-world area, wherein the surface is associated with an obstacle captured by a user via an image sensor in a mobile device (block 1110), and (e.g., in Figure 2 The method may include defining an AR generation area around the target location based on the surface (block 1130). In some implementations, the method may include defining an AR generation area around the target location via a user interface (e.g., in the example of FIG. 11 ). Figure 2 In some implementations, the method may include (for example, at Figure 2 The AR generation area engine 231 shown in FIG. 2 defines the AR generation area such that the AR generation area does not intersect with the surface.
[0082] like Fig.12 As shown, a method may include (for example, by Figure 2 The surface recognition engine 225 shown in the figure detects a surface in the real world area, wherein the surface is associated with an obstacle captured by the user via an image sensor in the mobile device (1210), and (e.g., in Figure 2 The method may include (for example, at the anchor point generation engine 227 shown in FIG. 1 ) receiving an AR generation instruction to generate an AR anchor point at a target location ( 1220 ). Figure 2 The AR generation area engine 231 shown defines an AR generation area around the target location based on the surface (1230).
[0083] Return to reference Figure 2 , the IMU 214 can be configured to detect motion, movement, and / or acceleration of the mobile device 110. The IMU 214 can include a variety of different types of sensors, such as accelerometers, gyroscopes, magnetometers, and other such sensors. The orientation of the mobile device 110 can be detected and tracked based on data provided by the IMU 214 or the GPS receiver 216.
[0084] The GPS receiver 216 can receive signals transmitted by GPS satellites. The signals include the time and location of the satellites. Based on receiving signals from several satellites (e.g., at least four), the GPS receiver 216 can determine the global location of the mobile device 110.
[0085] Other applications 240 include any other applications installed or otherwise available for execution on mobile device 110. In some implementations, application 222 can cause one of other applications 240 to be launched.
[0086] The device positioning system 242 determines the location of the mobile device 110. The device positioning system 242 can use the sensor system 210 to determine the position and orientation of the mobile device 110 within a global or physical space.
[0087] The AR anchor positioning engine 224 may include a machine learning module that can identify at least some types of entities within an image. For example, the machine learning module may include a neural network system. A neural network is a computational model for machine learning and is composed of nodes organized in layers with weighted connections. Training a neural network uses training examples, each of which is an input and a desired output, to determine the weight values of the connections between layers over a series of iterative rounds, which increases the likelihood that the neural network provides the desired output for a given input. During each training round, the weights are adjusted to account for incorrect output values. Once trained, the neural network can be used to predict outputs based on the input provided.
[0088] In some implementations, the neural network system includes a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one of the layers of the neural network is a convolutional layer. A convolutional layer is a layer in which the values of the layer are calculated based on applying a kernel function to a subset of the values of the previous layer. Training a neural network can involve adjusting the weights of the kernel function based on training examples. Typically, the same kernel function is used to calculate each value in a convolutional layer. Therefore, far fewer weights must be learned when training a convolutional layer than a fully connected layer in a neural network (e.g., a layer in which each value in a layer is calculated as a weighted combination of independent adjustments of each value in the previous layer). Because there are typically fewer weights in a convolutional layer, training and using a convolutional layer may require less memory, processor cycles, and time than an equivalent fully connected layer.
[0089] The communication module 206 includes one or more devices for communicating with other computing devices such as the AR server 252. The communication module 206 can communicate via a wireless or wired network such as the network 290. The communication module 256 of the AR server 252 can be similar to the communication module 206. The network 290 can be the Internet, a local area network (LAN), a wireless local area network (WLAN), and / or any other network.
[0090] The display device 208 may, for example, include an LCD (liquid crystal display) screen, an LED (light emitting diode) screen, an OLED (organic light emitting diode) screen, a touch screen, or any other screen or display for displaying images or information to a user. In some implementations, the display device 208 includes a light projector arranged to project light onto a portion of the user's eye.
[0091] The memory 220 may include one or more non-transitory computer-readable storage media. The memory 220 may store instructions and data that may be used by the mobile device 110 to implement the techniques described herein, such as generating a visual content query based on a captured image, transmitting a visual content query, receiving a response to the visual content query, and presenting a digital supplement identified in response to the visual content query. The memory 260 of the AR server 252 may be similar to the memory 220 and may store data instructions that may be used to implement the techniques of the AR server 252.
[0092] Processor component 204 and / or processor component 254 include one or more devices capable of executing instructions, such as instructions stored by memory 220, to perform various tasks. For example, one or more of processor components 204, 254 may include a central processing unit (CPU) and / or a graphics processor unit (GPU). For example, if a GPU is present, some image / video rendering tasks, such as generating and displaying portions of a user interface or displaying a digital supplement, may be offloaded from the CPU to the GPU. In some implementations, some image recognition tasks may also be offloaded from the CPU to the GPU.
[0093] Despite Figure 2 10. Although not shown in the drawings, some implementations include a head mounted display device (HMD) (e.g., AR enabled glasses). The HMD can be a separate device from the mobile device 110, or the mobile device 110 can include the HMD. In some implementations, the mobile device 110 communicates with the HMD via a cable. For example, the mobile device 110 can transmit video signals and / or audio signals to the HMD for display to the user, and the HMD can transmit motion, position, and / or orientation information to the mobile device 110.
[0094] The mobile device 110 may also include various user input components (not shown), such as a controller that communicates with the mobile device 110 using a wireless communication protocol. In some implementations, the mobile device 110 may communicate with an HMD (not shown) via a wired connection (e.g., a universal serial bus (USB) cable) or via a wireless communication protocol (e.g., any WiFi protocol, any Bluetooth protocol, Zigbee, etc.). In some implementations, the mobile device 110 is a component of the HMD and may be contained within a housing of the HMD.
[0095] Fig.13 Examples of a general purpose computer device 2000 and a general purpose mobile computer device 2050 that can be used with the techniques described herein are shown. Computing device 2000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, tablet computers, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device 2050 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are exemplary only and are not intended to limit implementations of the invention described and / or claimed in this document.
[0096] The computing device 2000 includes a processor 2002, a memory 2004, a storage device 2006, a high-speed interface 2008 connected to the memory 2004 and a high-speed expansion port 2010, and a low-speed interface 2012 connected to a low-speed bus 2014 and the storage device 2006. The processor 2002 can be a semiconductor-based processor. The memory 2004 can be a semiconductor-based memory. Each of the components 2002, 2004, 2006, 2008, 2010 and 2012 is interconnected using various buses and can be installed on a common motherboard or in other appropriate ways. The processor 2002 can process instructions for execution within the computing device 2000, including instructions stored in the memory 2004 or on the storage device 2006, to display graphical information for a GUI on an external input / output device (e.g., a display 2016 coupled to the high-speed interface 2008). In other implementations, multiple processors and / or multiple buses and multiple memories and memory types can be used appropriately. Furthermore, multiple computing devices 2000 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).
[0097] The memory 2004 stores information within the computing device 2000. In one implementation, the memory 2004 is one or more volatile memory units. In another implementation, the memory 2004 is one or more non-volatile memory units. The memory 2004 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.
[0098] The storage device 2006 can provide mass storage for the computing device 2000. In one implementation, the storage device 2006 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or a device array, including a device in a storage area network or other configuration. A computer program product can be tangibly implemented in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as a memory 2004, a storage device 2006, or a memory on a processor 2002.
[0099] The high-speed controller 2008 manages bandwidth-intensive operations of the computing device 2000, while the low-speed controller 2012 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one implementation, the high-speed controller 2008 is coupled to the memory 2004, the display 2016 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 2010 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 2012 is coupled to the storage device 2006 and the low-speed expansion port 2014. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices (such as keyboards, pointing devices, scanners) or networking devices (such as switches or routers), for example, through a network adapter.
[0100] As shown, computing device 2000 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 2020, or implemented multiple times in a group of such servers. It can also be implemented as part of a rack server system 2024. In addition, it can be implemented in a personal computer such as a laptop computer 2022. Alternatively, components from computing device 2000 can be combined with other components in a mobile device (not shown) such as device 2050. Each such device can contain one or more of computing devices 2000, 2050, and the entire system can be composed of multiple computing devices 2000, 2050 that communicate with each other.
[0101] The computing device 2050 includes a processor 2052, a memory 2064, an input / output device such as a display 2054, a communication interface 2066, and a transceiver 2068, among other components. The device 2050 may also be provided with a storage device such as a microdrive or other device to provide additional storage. Each of the components 2050, 2052, 2064, 2054, 2066, and 2068 are interconnected using various buses, and several components may be mounted on a common motherboard or in other suitable ways.
[0102] The processor 2052 can execute instructions within the computing device 2050, including instructions stored in the memory 2064. The processor can be implemented as a chipset including a separate chip with multiple analog and digital processors. The processor can provide, for example, coordination of other components of the device 2050, such as control of a user interface, applications executed by the device 2050, and wireless communications performed by the device 2050.
[0103] The processor 2052 can communicate with the user through the control interface 2058 and the display interface 2056 coupled to the display 2054. The display 2054 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 2056 may include appropriate circuits for driving the display 2054 to present graphics and other information to the user. The control interface 2058 can receive commands from the user and convert them to submit to the processor 2052. In addition, an external interface 2062 in communication with the processor 2052 can be provided so that the device 2050 can communicate with other devices in close proximity. The external interface 2062 can, for example, provide wired communication in some implementations, or provide wireless communication in other implementations, and multiple interfaces can also be used.
[0104] The memory 2064 stores information within the computing device 2050. The memory 2064 can be implemented as one or more of the following: one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 2074 can also be provided and connected to the device 2050 via an expansion interface 2072, which can include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 2074 can provide additional storage space for the device 2050, or can also store applications or other information for the device 2050. Specifically, the expansion memory 2074 can include instructions for executing or supplementing the above-mentioned process, and can also include security information. Therefore, for example, the expansion memory 2074 can be set as a security module for the device 2050, and can be programmed with instructions that allow the safe use of the device 2050. In addition, security applications and additional information can be provided via a SIMM card, such as placing identification information on the SIMM card in an inviolable manner.
[0105] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly implemented in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as the memory 2064, the expansion memory 2074, or a memory on the processor 2052, which may be received, for example, via the transceiver 2068 or the external interface 2062.
[0106] Device 2050 can communicate wirelessly via communication interface 2066, which can include digital signal processing circuitry when necessary. Communication interface 2066 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messages, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication can occur, for example, via radio frequency transceiver 2068. In addition, short-range communication can be performed, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, GPS (global positioning system) receiver module 2070 can provide additional navigation and location-related wireless data to device 2050, which can be used appropriately by applications running on device 2050.
[0107] Device 2050 may also communicate audibly using audio codec 2060, which may receive verbal information from a user and convert it into usable digital information. Audio codec 2060 may similarly generate audible sounds for the user, such as through a speaker in, for example, a microphone of device 2050. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 2050.
[0108] As shown, computing device 2050 can be implemented in many different forms. For example, it can be implemented as a cellular phone 2080. It can also be implemented as part of a smart phone 2082, a personal digital assistant, or other similar mobile device.
[0109] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system and to send data and instructions to the storage system.
[0110] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.
[0112] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (such as an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.
[0113] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
[0114] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the invention.
[0115] Additionally, the logic flows depicted in the accompanying drawings do not require the particular order shown or sequential order to achieve desired results. Additionally, other steps may be provided from the depicted flows, or steps may be eliminated, and other components may be added to or removed from the depicted systems.
Claims
1. A method, include: detecting a surface within a real-world area captured by a user via an image sensor in a mobile device; receiving, at an anchor generation engine, an AR generation instruction from the user to generate an augmented reality (AR) anchor that intersects the surface within the real-world area, and the AR anchor includes an AR generation area associated with the AR anchor, the AR anchor including a mapping of a physical area around at least a portion of a target location for displaying an AR object; as well as A capture instruction is defined in response to the AR generation instruction and based on the intersection, wherein the capture instruction defines the AR generation area.
2. The method according to claim 1, in, Defining the capture instructions includes modifying the capture instructions according to default capture instructions.
3. The method according to claim 1, in, The capture instruction includes a capture arc.
4. The method according to claim 3, in, The capture arc is less than 360 degrees.
5. The method according to claim 1, in, The capture instructions are displayed within the adaptive AR placement user interface.
6. The method according to any one of claims 1 to 4, in, The surface is a first surface, and the method further comprises: Detecting a second surface within the real world area, and defining the capture instructions to be based on the first surface and the second surface.
7. The method according to claim 6, in, The capture instructions include a capture arc of less than 180 degrees based on the first surface and the second surface.
8. A device, include: a sensor system configured to capture at least a portion of a real world area using a mobile device; a surface recognition engine configured to detect surfaces within the real world area; an anchor generation engine configured to receive an AR generation instruction from a user to generate an augmented reality (AR) anchor that intersects the surface within the real-world area, and the AR anchor includes an AR generation area associated with the AR anchor, wherein the AR anchor includes a mapping of a physical area around at least a portion of a target location, the AR anchor being generated at the target location within a threshold distance of the surface within the real-world area; as well as An adaptive AR placement user interface is configured to respond to the AR generation instruction and define a capture instruction based on the target position being within the threshold distance, the capture instruction defining the AR generation area.
9. The device according to claim 8, in, The acquisition instructions are modified from default acquisition instructions that define a scan completely around the target location.
10. The device according to claim 8 or 9, in, The capture instruction includes a capture arc of less than 360 degrees.
