Location recognition in augmented reality environments

By generating and sharing relative description files, the device realizes continuous updates of the positioning and virtual objects in an augmented reality environment without direct line of sight to physical markers, solving the instability and coordinates of the device positioning in the prior art The issue of recentering has improved the stability of the augmented reality environment and the ability of multi-device collaborative work.

CN111373410BActive Publication Date: 2024-11-05GOOGLE LLC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN201880075152.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-11-22
Filing Date
2018-11-19
Publication Date
2024-11-05
Estimated Expiration
2038-11-19

AI Technical Summary

Technical Problem

In existing augmented reality environments, it is difficult for devices to maintain the exact positioning and coordinate system when they do not directly look at the physical marker, especially in multi-device scenarios, resulting in the discontinuity of presentation of virtual objects and the recentering of the coordinate system.

Method used

By generating and sharing relative description files (RDFs), devices can capture the relative location of markers through image sensors without direct line of sight to physical markers, initially locate and identify markers, update the location of virtual objects, and ensure an augmented reality environment without direct sighting. 的连续性和多设备间的同步。

Benefits of technology

It realizes the accurate positioning of the device when the line of sight is not directly targeted to the mark and the continuous update of the virtual object, solves the problem of recentering the coordinate system, and improves the stability of the augmented reality environment and the ability of multi-device collaborative work.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111373410B_ABST
    Figure CN111373410B_ABST
Patent Text Reader

Abstract

A method includes: receiving a relative description file of physical markers placed at a location in a first device, the relative description file defining the relative position of each of the physical markers with respect to at least another of the physical markers; initially locating the position of the first device among the physical markers by visually capturing any first physical marker among the physical markers using an image sensor of the first device; and identifying a second physical marker among the physical markers and the position of the second physical marker without line of sight, the second physical marker being identified using the relative description file.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and is a continuation of U.S. Non-Provisional Patent Application No. 15 / 820,808, filed on November 22, 2017, entitled “POSITIONAL RECOGNITION FORAUGMENTED REALITY ENVIRONMENT,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This document generally relates to location identification for augmented reality environments. Background Art

[0004] In the context of computer-based consumption of media and other content, it is becoming increasingly common to provide participants with immersive experiences. One area involves presenting virtual reality (VR) and / or augmented reality (AR) environments on a device such as a smartphone or tablet. In an AR environment, a person can view a screen that presents at least aspects of a physical environment (e.g., a video or image of a physical space) and aspects of VR (e.g., virtual objects superimposed on the video or image). Summary of the invention

[0005] In a first aspect, a method includes: receiving a relative description file of physical markers placed at a location in a first device, the relative description file defining a relative position of each of the physical markers with respect to at least another of the physical markers; initially positioning the first device among the physical markers by visually capturing any one of the first physical markers using an image sensor of the first device; and identifying a second physical marker among the physical markers and the position of the second physical marker without line of sight, the second physical marker being identified using the relative description file.

[0006] Implementations may include any or all of the following features. At least the first physical marker may include a code. At least the first physical marker may include a natural marker. The method may further include presenting an augmented reality environment on a display of the first device, the augmented reality environment including an image captured using at least a camera of the first device and at least a first virtual object applied to the image. The first virtual object may be positioned in the image at a predefined position relative to the first physical marker. The position of the first device may be initially located by aligning the camera in a first direction toward the first physical marker, and after the position of the first device is initially located, the user may align the camera in a second direction, and identifying the second physical marker may include using a relative description file to determine that the position of the second physical marker is located in the second direction. The method may further include updating the augmented reality environment with the second virtual object at the predefined position relative to the second physical marker. The user may move the first device to the position of the second physical marker, and at the position of the second physical marker, the first physical marker may no longer be visible in the augmented reality environment, and the user may align the camera in a third direction toward the position of the first physical marker without line of sight, and the method may further include updating the augmented reality environment with the first virtual object at the predefined position relative to the first physical marker. Before the second physical marker is identified, the second physical marker may be removed from the position of the second physical marker, and the position of the second physical marker and the second physical marker may be identified using a relative description file. Initially locating the position of the first device may include centering the coordinate system of the first device based on the position of the first physical marker, wherein identifying the second device physical marker does not include re-centering the coordinate system. Centering the coordinate system of the first device may include recalibrating the center of the coordinate system to correct tracking errors. The method may further include providing a relative description file to the second device, wherein the second device is configured to initially locate the position of the second device among the physical markers by visually capturing any one of the physical markers using an image sensor of the second device, and using the relative description file to identify the positions of all remaining physical markers in the physical markers without line of sight. The relative description file may be a first relative description file, and the method may further include receiving a second relative description file from the second device, and at least partially merging the first relative description file and the second relative description file. The relative description file may define the relative position of the second physical marker with respect to the first physical marker, the relative position including the displacement of the second physical marker relative to the first physical marker. The first device may recognize at least some of the physical markers, and the method may further include generating a log on the first device reflecting at least some of the physical markers that the first device has recognized.The relative description file may define the relative position of each of the physical tags with respect to all other physical tags in the physical tags.

[0007] In a second aspect, a non-transitory storage medium may have instructions stored thereon, which when executed are configured to cause a processor to perform operations, the operations including: receiving a relative description file of physical markers placed at locations in a first device, the relative description file defining the relative position of each of the physical markers with respect to at least another of the physical markers; initially locating the position of the first device among the physical markers by visually capturing any one of the first physical markers using an image sensor of the first device; and identifying a second physical marker among the physical markers and the position of the second physical marker without line of sight, the second physical marker being identified using the relative description file.

[0008] In a third aspect, a system includes: a first device, the first device being configured to receive a location point determined using the first device for each physical marker among physical markers placed at a location, each of the location points being associated with a corresponding one of the physical markers, the first device being further configured to generate a relative description file of the physical markers based on the location points, the relative description file defining the relative position of each of the physical markers with respect to at least another one of the physical markers; and a second device and a third device, the second device and the third device being both configured to receive the relative description file generated by the first device, wherein each of the second device and the third device is further configured to initially locate the position of the second device or the third device among the physical markers by visually capturing any one of the physical markers using an image sensor of the second device or the third device, respectively, and to identify the positions of all remaining physical markers among the physical markers using the relative description file without line of sight.

[0009] In a fourth aspect, a method includes: receiving, in a first device, a location point determined using the first device for each of the physical markers placed at a location, each of the location points being associated with a corresponding one of the physical markers; generating a relative description file of the physical markers based on the location points, the relative description file including a relative position of each of the physical markers with respect to at least another of the physical markers; and providing the relative description file to at least a second device and a third device, wherein each of the second device and the third device is configured to initially locate the position of the second device or the third device among the physical markers by visually capturing any one of the physical markers using an image sensor of the second device or the third device, respectively, and identifying the positions of all remaining physical markers among the physical markers using the relative description file without line of sight.

[0010] Implementations may include any or all of the following features. At least one of the physical markers may include a code. At least one of the physical markers may include a natural marker. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example of a system that can be used to generate an immersive experience in an augmented reality (AR) environment is shown.

[0012] Figure 2A -D shows an example of a device with an AR environment.

[0013] Figure 3 An example of devices sharing relative description files and merging them with each other is shown.

[0014] Figure 4 An example of a method involving generation of a relative description file (RDF) is shown.

[0015] Figure 5 An example of a method involving the use of RDF to identify tags is shown.

[0016] Figure 6 An example of a method involving at least one device sharing an RDF and merging the RDF with a received RDF is shown.

[0017] Figure 7 Examples of computer devices and mobile computer devices according to the disclosed embodiments are shown.

[0018] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0019] This document describes examples of identifying markers and their relative positions and orientations in an augmented reality (AR) environment. The AR environment may include images of physical reality (e.g., a camera view around a user) and images of virtual reality (e.g., AR objects). The presentation of the AR environment may then provide a view to the user that simultaneously displays at least some of the images of physical reality and at least some of the images of virtual reality. In some embodiments, a relative description file (RDF) may define relative positions between a particular set of physical markers that have been placed at respective locations. Such an RDF may be relatively limited in size and easily shared with multiple devices. A device with the RDF may locate its own position by identifying one of the physical markers (e.g., using a camera), extrapolate the device's position and orientation, and may use the RDF to learn and act based on the positions and identifications of other physical markers when those markers are not currently visible or cannot be directly detected by the device. Some implementations may identify the relative positions and orientations of physical markers, while some examples herein only mention identification of positions for the sake of brevity. Some embodiments may also or alternatively be applied to virtual reality systems, for example, to provide an immersive virtual reality experience.

[0020] Existing systems for handling AR environments have attempted to use unique and identifiable items (such as QR codes) in the physical environment as anchors for virtual objects. However, such systems may depend on the availability of sight to the items. As another example, after the first such marker is identified, the identification of a second (or subsequent) marker can trigger the re-centering of the applied coordinate system. When the device no longer sees the marker, it may lose the context of the location because such a system may not be able to extrapolate the location in the world after the marker is no longer visible. That is, the system is typically configured to treat the captured marker as the "world center" of the device for the purpose of presenting the AR environment, so that the AR coordinate system is centered at the location where the marker is located. One disadvantage of this approach is that if more than one marker is used, in a sense, whenever the device captures the next marker, the device re-centers its coordinate system. Another disadvantage is that when multiple devices are involved, it may become challenging to have a common understanding of where the center of the world is.

[0021] Some previous systems have attempted to use area description files (ADFs) to facilitate the use of markers in AR contexts. Creating an ADF can be a cumbersome process, as it may be necessary to walk around the room with the device and have the camera obtain details of all the structures and objects present. The approach is based on creating a comprehensive understanding of the physical location in a preliminary stage. Depending on the size of the physical space and the complexity of the structures therein, the resulting ADF may be relatively large. In multi-device scenarios, where several devices should be using the same ADF in order to facilitate a common AR experience for users, distributing a large ADF to all devices may take some time and may impose significant demands on the communication network.

[0022] Recently, some AR systems have taken a slightly different approach to device orientation. For example, some systems are configured to perform environment tracking using multiple lenses on a handheld device such as a smartphone or tablet. The system can use two lenses and infrared signals to effectively triangulate its position within a room. This can allow a Tango system to have stereo vision and form a stereoscopic understanding of the room, including its depth and what surfaces and objects are in the room. The system can conveniently map these features to facilitate its spatial understanding. Other systems use similar techniques to provide AR functionality.

[0023] One feature of some AR systems is the ability to extrapolate from the position of the sensing device in the room. The system can rely on an anchor point in the physical world to perform continuous extrapolation of position to expand its understanding of the structure of the space from that anchor point. One benefit of this is that the system does not rely on having a line of sight to a known marker. Instead, by extrapolating from the current position of the device, the system can infer the position of the marker and can perform AR functions based on it, such as placing a virtual object at a location corresponding to the marker.

[0024] In some embodiments, the system can use physical markers for at least two beneficial purposes: indicating a location in space, for example, to define the world center of the device; and using one or more markers as real-world clues to the audience about where they can expect AR objects to appear. In some embodiments, organizers of educational conferences such as lectures or tours can place markers in a room or other physical location. The organizer can capture each marker using a handheld device so that it is defined for the AR system which marker is located where. When attendees capture the markers using their own devices, the selected virtual objects can become visible in the AR environment presented on the attendees' respective devices. To name just two examples, the virtual objects can all be substantially the same as each other, or two or more of the virtual objects can be different from each other. The virtual objects can have any appearance. In some embodiments, one or more virtual objects can include content such as advertisements. Based on knowing the relative positions of the markers relative to each other, the AR system can display the virtual object corresponding to the marker to the attendee before the attendee's device directly detects the marker. Examples are described below.

[0025] Figure 1 An example of a system 100 is shown that can be used to generate an immersive experience through an AR environment. In some embodiments, the immersive experience allows a user to see one or more AR objects in conjunction with an image of physical reality. For example, the immersive experience can be organized in the form of a tour, which includes multiple AR objects placed at corresponding positions relative to physical reality. The system 100 is shown here in a space 102, which can be any kind of physical space, including but not limited to a room or other indoor or outdoor premises. The space 102 is schematically shown here from above. Tables 104A-B in the space 102 are visible. For example, the organizer of the tour may be planning that participants should visit several locations on the tables 104A-B, and there are AR experiences presented as part of the tour. Any other physical structure other than the table may be used instead or in addition.

[0026] The organizer may place multiple markers in space 102. Here, markers 106A and 106B have been placed on top of table 104A, and markers 106C and 106D have been placed on top of table 104B. For example, the organizer may have decided that markers 106A-D should be visited in alphabetical order from 106A to 106D, or they may be visited in a random order. Markers 106A-D may be used to trigger the appearance of and define locations for one or more virtual objects that will be visible to tour participants, for example, as will be described below.

[0027] A participant positioned at one of the markers 106A-D may have a line of sight to one or more other markers. Line of sight, when available, may allow the participant and / or the participant's device to view (e.g., directly view in a straight line) the other markers from their current position. However, in some cases, line of sight is not available. For example, a structure such as a wall 108 may be positioned between two or more markers. As another example, a participant and the participant's device may view the other markers from a particular viewing point (perspective) that is currently difficult or impossible for the other markers to perceive from the current position.

[0028] The organizer can use one or more devices to create an AR experience. Here, a device 110 is schematically shown in space 102. Any type of device can be used, including but not limited to a smart phone or a tablet device. Device 110 includes an AR system 112 here. The AR system uses light sensing (e.g., infrared light) and two or more lenses to generate stereoscopic vision in space 102 in order to form its stereoscopic viewing understanding, which allows the AR system 112 to also extrapolate to the position of space 102 where there is currently no line of sight. For example, the AR system 112 can be an AR system or part thereof, which uses a sensing device and is configured to use a positioning point in the physical world to extrapolate from the position of the sensing device in the room. Such an AR system can perform continuous extrapolation of the position to expand its recognition of the spatial structure from the positioning point. In some embodiments, the AR system 112 is executed partially on the device 110 and partially on one or more other devices (e.g., another handheld device or a cloud system).

[0029] The device 110 may have one or more display screens 114. In some embodiments, the display screen may be a touch screen. For example, the display screen 114 may present a control 116 that allows the organizer to create a tour using the device 110 and (here) the markers 106A-D. Having activated the AR system 112 using the control 116, the organizer may visit the distributed markers 106A in any order. Here, for example, the organizer visits locations 118A to 118D in that order to capture the corresponding markers 106A-D. The device may capture the corresponding marker at each location and record its location. For example, the organizer may use a control 120 that may be presented on the display screen 114 to trigger the capture. As another example, the device 110 may continuously monitor the presence of markers such as markers 106A-D from signals from its sensors, and may capture such markers whenever detected. When the device captures any one of the markers 106A-D, the AR system 112 may register the identity of the marker and the location of the marker. For example, such registration may reflect here that marker 106A is positioned at position 118, marker 106B is positioned at position 118B, marker 106C is positioned at position 118C, and marker 106D is positioned at position 118D. The position of any or all of the markers may be defined relative to a known position, such as relative to an anchor marker or other positioning point, for example as will be described below.

[0030] The initial positioning of device 110 (relative to any of markers 106A-D in this example) may not necessarily mean that this particular marker is subsequently defined as (0,0,0) of the AR world. Instead, the positioning of a marker may be defined by an offset relative to the center of the world defined by the RDF. Such a center may be established by earlier positioning (e.g., by a person placing / defining markers in a physical environment) or by definition (such as by selecting the North Pole).

[0031] The AR system 112 may generate one or more data structures based on the capture of the markers 106A-D performed with the device 110. Here, a relative description file (RDF) 122 is generated. The RDF 122 defines the relative position of each of the markers 106A-D with respect to at least one other of the markers 106A-D. In some implementations, the marker 106A may be considered an anchor marker. The RDF 122 may define the relative position of the marker 106B based on the detected position of the marker 106A; may define the relative position of the marker 106C based on the detected position of the marker 106B; and may define the relative position of the marker 106D based on the detected position of the marker 106C. For example, if the position of the marker 106A is defined as (0,0,0) using a Cartesian coordinate system for the space 102, the relative position of the marker 106B may be defined as a displacement (50,100,0) relative to the position of the marker 106A. In some implementations, the RDF 122 defines the relative position of each of the tags 106A-D with respect to all other tags 106A-D.

[0032] In some embodiments, the user may begin by pointing the device 110 at one of the markers 106A-D. The device 110 may recognize one of the markers 106A-D and store the position of the marker based on the current position and / or orientation of the device 110 (e.g., in the RDF 122). Based on the device 110 recognizing one of the markers 106A-D, one or more virtual objects may be presented on the display screen 114 in association with the position of one of the markers 106A-D (e.g., on top of the marker). This presentation may be characterized as showing the user an experience. The user may move around the space 102 with the device 110. At some point, the device 110 (e.g., using a camera or other image sensor of the device 110) may find another of the markers 106A-D. The device 112 may then update its stored information (e.g., edit the RDF 122) to reflect the information about the new marker. For example, the new marker may be defined as having at least a position and rotation (e.g., orientation) relative to the marker that was first recognized. That is, at this point the RDF 122 may contain definitions of both markers. Similarly, one or more additional tags may be identified and RDF 122 updated accordingly.

[0033] The markers 106A-D can be recognized by the sensor on the device 110, so that the markers 106A-D can be captured accordingly. One or more suitable forms of markers can be used. In some embodiments, codes can be used. The code can be a digital code. For example, the marker can include a QR code detected by a light sensor (e.g., a camera) on the device 110. In some embodiments, non-code markers can be used. For example, a natural marker can be any relatively flat image with a sufficient number of features, rich contrast and / or a unique shape. Natural markers can be detected by a light sensor (e.g., a camera) on the device 110. In some embodiments, a marker can include one or more features that appear in a physical environment. Such a marker can be defined without physical intervention in the physical environment. Objects located in the physical environment can be defined as markers. For example, natural markers can be used, including but not limited to unique photos with high contrast edges or geometric designs with unique edges. Real-world markers can be used, including but not limited to images captured by photos of textures on a table, which can be used similarly to fingerprints. Through at least natural markers and real markers, the system looks for a set of image-based features. In some embodiments, these marker codes can be more accurately converted to bits and bytes. Based on the captured image, the associated tag identity may be determined.

[0034] In this way, the system 100 can be used to perform one or more processes related to marker definition and / or identification. For example, the device 110 can receive a location point for each of the markers 106A-D corresponding to the location 118A-D. The location point can be used to generate the RDF 122. The RDF 122 can define the relative position of each of the markers 106A-D with respect to at least another of the markers 106A-D. Such RDF 122 can be provided to one or more devices for providing an AR experience, such as will be described below.

[0035] In some embodiments, multiple AR entities can be represented in separate groups organized by corresponding physical markers. For example, users with different interests can explore such groups in different orders from each other. In some embodiments, teachers using the embodiments can divide users into groups among tables in the classroom for space management. Teachers can move groups between tables to inspect new AR objects or AR environments. For example, each of the user groups can remain in a corresponding area of ​​the classroom, and then can take turns to teach the remaining users about the AR objects that their group has explored. Users in other groups placed elsewhere in the classroom can aim their devices at the displayed group and see the relevant AR objects.

[0036] Figure 2A -D shows an example of a device with an AR environment. Figure 2A , a device 200 is shown, such as a handheld device having at least one display screen 202. The device 200 may be connected to the device 110 ( Figure 1 ). The AR environment 204 can be presented on the display screen 202. The AR environment 204 can include at least one image 206 captured using a camera of the device 200. For example, portions of the floor 208 and the desktop surface 210 are visible in the image 206. In some embodiments, the desktop surface 210 can correspond to the table 104A ( Figure 1 ). For example, the AR environment 204 here may reflect that the marker 106A has been captured. The marker 106A is a physical object that can be detected by the camera and is visible here in the image 206 of the AR environment 204. The image 206 can be a still image, or can be a continuous feed of essentially real-time video captured by the camera of the device 200. The AR environment 204 may include at least one virtual object 212 applied to the image 206. Here, the object 212 is an apple. For example, the apple is an AR element that has been assigned to the marker 106A. The object 212 is placed in a position in a predefined position relative to the marker 106A. In this example, the object 212 has been applied to the desktop surface 210, but may be applied alternatively or additionally to any other part or aspect of the image 206, such as by moving the AR element within the AR environment 204.

[0037] The device 200 includes an AR system 214 as schematically indicated. For example, the AR system can be connected to the AR system 112 ( Figure 1 ) is the same or similar. AR system 214 can receive, maintain, and use one or more RDFs 216. RDF 216 can be the same as RDF 122 ( Figure 1 ) are similar or identical. In some embodiments, a device (e.g., Figure 1 Device 110 in the example may provide RDF 216 to one or more devices such as device 200. Device 200 may be configured to initially locate its position at a plurality of markers (e.g., Figure 1 For example, device 200 may include an image sensor (e.g., a camera) and may use the image sensor to visually capture any or all of the markers. Device 200 may also be configured to use RDF 216 to identify all remaining markers and their locations without line of sight. For example, when device 200 is positioned at one of markers 106A-D ( Figure 1 ), the RDF 216 will also inform the AR system 214 of the identities and locations of all other markers in the markers 106A-D, also when the device 200 does not have a direct line of sight to any such markers.

[0038] The AR system 214 can use one or more sensors 218 of the device 200 to capture the marker. In some embodiments, a light sensor can be used. For example, a sensor for infrared light can use two lenses on the device 200 to obtain a stereoscopic view of the physical location.

[0039] The signals from the sensors 218 may be interpreted by one or more image analyzers 220. For example, the image analyzer 220 may read a visual symbol including a QR code and notify the AR system 214 of the information encoded in the symbol. As another example, the image analyzer 220 may capture an image of a natural marker and compare at least a portion of the captured image to an image-to-identifier mapping to determine the identity of the captured natural marker.

[0040] RDF 216 can allow a user to experience various aspects as part of a tour defined by distributed markup. For example, after a user locates device 200, such as by capturing Figure 2A , the user can point the sensor 218 (e.g., a camera or other light sensor) in a certain direction, and the system will indicate whether any markers are placed along that direction. For example, assuming and again referring to Figure 1 , the user stands at position 118A and aligns the sensor of device 200 at position 118C where marker 106C is located. The line of sight may then be blocked by wall 108. That is, device 200 does not have a line of sight from position 118A toward position 118C. Figure 2B An example of what display screen 202 of device 200 may present in this case is shown. In AR environment 204, updated image 206' shows that wall 108 substantially blocks the line of sight in the current direction. Table 104B is obscured by wall 108, but is indicated with ghosting for clarity.

[0041] The AR system 214 of the device 200 can access the RDF 216, which defines the relative positions of the available markers. Here, the device is currently placed at the position 118A and is aimed at the direction of the table 104B. Then, the RDF 216 can indicate that the marker 106C is placed in this direction based on the current position of the device 200 and the current direction of the sensor. That is, the AR system 214 can extrapolate beyond the wall 108 and can determine that the marker 106C will be in the current line of sight if there is no wall 108. Therefore, the AR system 214 can recognize the marker 106C in the current situation, although the marker 106C is not currently visible to the device. Based on the recognition of the marker 106C, the AR system 214 can place one or more virtual objects 222 in the AR environment 204. For example, the virtual object 222 is a banana here, which is added to the image 206' in the virtual environment 204. That is, after the position of the device 200 is initially located at the position 118A using the marker 106A, the user can aim the camera of the device 200 in any direction. Then, by using RDF 216 to determine that the relative position of marker 106C with respect to marker 106A is located at the current orientation, another marker (eg, marker 106C) may be identified.

[0042] In some implementations, the AR system 214 can identify the marker 106C and its location in the above situation, regardless of whether the marker 106C is still at the location. That is, the identification can be based on the RDF 216, which defines the location 118C relative to the location 118A where the device 200 is currently located in this example. In this way, the AR system can identify the location 118C as the location where the marker 106C was initially captured based on the information in the RDF 216, and can present the virtual object 222 in the AR environment 204 based on the identification. That is, the marker 106C can be removed from the location 118C and can still be identified by the AR system 214 based on the RDF 216 thereafter.

[0043] The AR system 214 can log one or more events about the device 200. In some implementations, the AR system 214 can generate a log on the device 200 that reflects the markers 106A and 106C that the device 200 has recognized in this example. Figure 1 If one or more of the tags 106B and 106D in the , this can be reflected in the log.

[0044] RDF 216 may help identify any tags whose relative locations are defined in RDF 216. For example, suppose the user moves to location 118C ( Figure 1) and point the camera of device 200 in a direction toward location 118A. RDF 216 may then indicate that marker 106A is positioned in this direction based on the current location of device 200 and the current orientation of the sensor. Figure 2C 106 ” is shown as having been added to the image 206 ” of the AR environment 204 presented on the display screen 202 . That is, even though the table 104A (schematically indicated here) is obscured by the wall 108 , the AR system 214 can still extrapolate this obstruction beyond the line of sight, identify the invisible marker 106A and place the virtual object 212 in the correct location. For example, if the device 200 had lost its orientation for any reason while the device 200 was positioned at the location 118C, the user can reposition the device 200 relative to the marker 106C and thereafter point the camera at another location, such as towards the location 118A. Because the device 200 is positioned again in this example, and because the RDF 216 indicates the location of the virtual object relative to the device 200 , the system can place the virtual object 212 so that it is visible to the user from the current location, even though the device 200 is not currently detecting the marker 106A ( Figure 2A )itself.

[0045] The initial positioning of the device at the marker 106A, such as in the position 118A, can correspond to the centering of the coordinate system of the device 200 at that position. For example, the AR system 214 can define the coordinate system based on the readings from one or more sensors 218. In addition, if the device 200 is moved to the position 118D, when the AR system 214 subsequently detects another marker such as the marker 106D, the AR system 214 may not re-center the coordinate system. Instead, the AR system 214 can identify the marker 118D based on the relative position defined in the RDF 216, and can place the virtual object in the position based on the identified marker without re-centering its coordinate system when doing so. Although the coordinate system is not re-centered, the center of the coordinate system can be recalibrated. For example, this can be done to correct tracking errors. In some embodiments, a subsequent marker (e.g., marker 106C) can be used to re-position the device 200 so that any tracking errors when the device 200 is re-positioned from, for example, marker 106A to marker 106C are corrected.

[0046] In some embodiments, the location of the device 200 can be initially located by centering the device's coordinate system based on the location of the physical marker. Another physical marker can then be used to re-center the coordinate system. In doing so, any virtual objects (e.g., virtual objects 212, 222) can continue to be presented in the same space. In some embodiments, this can occur in a camera passthrough manner, for example because the relative coordinate scale is the same. In this way, the system can be re-centered but the virtual objects can remain in the same position. That is, the entire virtual world can be displaced, but the relative positions of the markers can remain the same.

[0047] RDF 216 may be shared with one or more devices. For example, Figure 2D 2 shows a device 224 presenting an AR environment 228 on its display screen 226, the AR environment 228 currently including an image 230 having a virtual object 232 applied thereto. The device 224 may be similar to or the same as the device 200. For example, the device 224 visits the location 118B ( Figure 1 ) and recognizes marker 106B there, wherein virtual object 232 is triggered by the recognition of the marker. Device 224 can initially locate its position in space 102 ( Figure 1 ). For example, presentation of virtual object 232 may be triggered by positioning of device 224 at location 118B, or device 224 may have previously been positioned at another of the markers and may thereafter identify marker 118B based on RDF 216 having been received. That is, two devices with the same RDF (e.g., RDF 216) may position their locations at the same or different markers and then have a common spatial understanding of the space of distributed markers. To name just two examples, presentation of virtual object 232 may be triggered by positioning of another participant device (e.g., device 200) or an organizer's device (e.g., Figure 1 Device 110 in the example shares RDF 216 with device 224.

[0048] Figure 3An example of devices that share RDFs and merge them with each other is shown. Here, physical tags 300A-C are already distributed in a physical space. Here, RDF 302A is used by a first device (e.g., stored on the first device), and RDF 302B is used by a second device (e.g., stored on the second device). The first device in this example only visits tags 300A and 300B, but not tag 300C. For example, the first device locates its position at tag 300A, and thereafter captures tag 300B, or vice versa. Therefore, RDF 302A defines a relative position 304A connecting tags 300A-B. For example, relative position 304A may include a displacement of tag 300B relative to tag 300A. In contrast, RDF 302A does not define any relative position about tag 300C that has not yet been captured by the first device. In fact, in this case, tag 300C may not exist in RDF 302A, but is shown here for illustrative purposes.

[0049] In addition, in this example, the second device only visits tags 300B and 300C, but not tag 300A. For example, the second device locates its position at tag 300B, and thereafter captures tag 300C, and vice versa. Therefore, RDF 302B defines a relative position 304B connecting tags 300B-C. For example, relative position 304B may include the displacement of tag 300C relative to tag 300B. On the contrary, RDF 302B does not define any relative position about tag 300A that has not been captured by the second device. In fact, tag 300A may not exist in RDF 302B, but is shown here for illustrative purposes. Either or both of the first and second devices can share its corresponding RDF 302A or 302B with another device, such as with another of the first and second devices.

[0050] The device receiving the RDF may at least partially merge the received RDF with one or more RDFs known to the device. Here, operation 306 schematically illustrates that RDF 302A-B is merging with each other to form RDF 302C. RDF 302C reflects the relative positions of all tags 300A-C. That is, RDF 302C includes both relative positions 304A and 304B. From the perspective of the first device that has initially only identified the relative positions of tags 300A-B, RDF 302C corresponds to the increased knowledge of the relative positions of tag 300C. Similarly, from the perspective of the second device that has initially only identified the relative positions of tags 300B-C, RDF 302C corresponds to the increased knowledge of the relative positions of tag 300A. That is, the first device can understand the tag by sharing RDF 302B from the second device that has captured tag 300C, because the second device has also captured the tag 300B that the first device already knows.

[0051] The following examples relate to one or more of the RDFs described herein (e.g., RDFs 122, 216, 302A, 302B, and / or 302C). In the examples, reference is made to a system (e.g., Figure 1 The RDF version is described by the corresponding state of the system 100 in FIG. 2 and / or the AR system 214 in FIG. 2. Any suitable programming language can be used to implement the examples. Here, JavaScript Object Notation is used for illustration purposes.

[0052] In "state 1" of the system, the RDF may contain at least the following:

[0053]

[0054] The RDF in state 1 reflects that "Marker1" has been associated with a position corresponding to the x,y,z-coordinate (0,0,0). The RDF also has a "relativeTo" variable that defines the reference coordinate system or reference point of Marker1. Here, the value of the "relativeTo" variable is "self", indicating that the position of Marker1 is relative to its own position (0,0,0).

[0055] Now suppose the system recognizes another token that can trigger the system to enter another state and update the RDF accordingly. In "state 2" of the system, the RDF may contain at least the following:

[0056]

[0057] The RDF in state 2 reflects that Marker1 retains its position relative to itself corresponding to the x,y,z-coordinate (0,0,0), and that "Marker2" has been associated with the x,y,z-coordinate (1,2,3). For Marker2, the "relativeTo" variable has the value "Marker1", indicating that Marker2 has its position - i.e., (1,2,3) - defined relative to the position of Marker1.

[0058] Now suppose the system recognizes another marker, which can trigger the system to enter another state and update the RDF accordingly. For some reason, the system cannot immediately relate the new marker to any existing markers (Marker1 and Marker2 in this example). This may happen, for example, if the device loses its track after recognizing the previous marker. As another example, this may happen because a new user has joined and will update the common RDF based on recognizing one or more new markers. In "State 3" of the system, the RDF may contain at least the following:

[0059]

[0060] The RDF in state 3 reflects that Marker1 retains a position relative to itself corresponding to x,y,z-coordinates (0,0,0), Marker2 has been associated with x,y,z-coordinates (1,2,3) relative to the position of Marker1, "Marker3" has a position (0,0,0) defined relative to itself (i.e., the value of the "relativeTo" variable "Self"), and "Marker4" has a position (5,6,7) relative to the position of Marker3.

[0061] RDF can "self-heal" in one or more aspects. In some embodiments, RDF can self-heal to resolve situations such as in state 3 above, where there are multiple location definitions associated with itself (i.e., for tag 1 and tag 3 in this example). For example, if the device successfully tracks between items in each group, the system can self-heal such RDF. In "state 4" of the system, RDF can contain at least the following:

[0062]

[0063] The RDF in state 4 reflects that Marker1 retains its position relative to itself corresponding to the x,y,z-coordinate (0,0,0), Marker2 has been associated with the x,y,z-coordinate (1,2,3) relative to the position of Marker1, Marker3 has a position (10,11,12) defined relative to Marker1 and Marker4 has a position (15,16,17) relative to the position of Marker1. That is, Marker2, Marker3 and Marker4 now all have their positions defined relative to the position of Marker1.

[0064] Figure 4-6 Method examples are shown. Any or all operations of the method may be performed using one or more processors that execute instructions stored in a non-transitory storage medium, such as described below. In some embodiments, more or fewer operations than shown may be performed. In some embodiments, two or more operations may be performed in a different order. For illustrative purposes only, some components introduced elsewhere in this disclosure will be mentioned in the following description.

[0065] Figure 4 An example of a method 400 involving generating a relative description file (RDF) is shown. At 410, a tag can be placed. For example, an organizer can place tags 106A-D in space 102. At 420, the tag can be captured. For example, device 110 can capture tags 106A-D at corresponding locations 118A-D. At 430, an RDF can be generated. For example, RDF 122 can be generated for tags 106A-D. At 440, the RDF can be provided to one or more devices. For example, device 110 can share RDF 122 with device 200 and with device 224. Devices 200 and / or 224 can then use the provided RDF to locate their location and identify one or more tags.

[0066] At 450, the marker placement can be updated. For example, the organizer can change the location of one or more of the markers 106A-D, or can add one or more new physical markers. At 460, the updated markers can be captured. For example, the organizer can use device 110 to capture the updated markers in space 102. At 470, an updated RDF can be generated. For example, device 110 can update RDF 122. At 480, the updated RDF can be provided to one or more devices. For example, device 110 can share the updated RDF 122 with device 200 and with device 224. Devices 200 and / or 224 can then use the provided updated RDF to locate their locations and identify one or more markers.

[0067] Figure 5 An example of a method 500 involving marker recognition using RDF is shown. At 510, RDF may be received. For example, device 200 may receive RDF 216. At 520, positioning may be performed in comparison to physical markers. For example, device 200 may locate its position by capturing any one of markers 106A-D. At 530, a virtual object (VO) may be displayed in an AR environment. For example, device 200 may present virtual object 212 in AR environment 204. At 540, the device may move to the next marker. For example, device 200 may be moved from position 118A to position 118C. At 550, the device may identify the next marker. For example, the device may identify marker 106C at position 118C. Device 200 may identify marker 106C by capturing an image of marker 106C via sensor 218 on device 200 or by using an extrapolation of RDF 216 to indicate marker 106C to device 200. At 560, the user may aim the camera of the device in a direction. For example, when at position 118C, device 200 may be aligned toward position 118A. At 570, a marker may be identified. For example, device 200 may identify marker 106A at position 118A when at position 118C because a line of sight to marker 106A is available, or by way of RDF 216. At 580, one or more markers may be registered in a log. For example, device 200 may log markers 106A and 106C.

[0068] Figure 6 An example of a method 600 is shown, which involves at least one device sharing an RDF and merging the RDF with a received RDF. At 610, an RDF can be generated. For example, regarding Figure 3 The second device mentioned may generate RDF 302B. At 620, the RDF may be shared with at least one device. For example, the second device may share the RDF with the at least one device. Figure 3 The first device mentioned shares RDF 302B. At 630, an RDF can be received. For example, the second device can receive RDF 302A from the first device. At 640, two or more RDFs can be at least partially merged. For example, the second device can merge RDF 302A-B to form RDF 302C. That is, the second device can learn the tag 300A through RDF 302A shared from the first device that has captured the tag because the first device has also captured the tag 300B that the second device already knows.

[0069] Figure 7 Examples of computer devices and mobile computer devices are shown that can be used to implement the techniques described herein. Figure 7Examples of a general purpose computer device 700 and a general purpose mobile computer device 750 that can be used with the techniques described herein are shown. Computing device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, tablet computers, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are meant to be exemplary only and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0070] The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. The processor 702 can be a semiconductor-based processor. The memory 704 can be a semiconductor-based memory. Each of the components 702, 704, 706, 708, 710 and 712 uses various buses to interconnect and can be installed on a common motherboard or installed in other ways as appropriate. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or on the storage device 706 to display graphical information of a GUI on an external input / output device such as a display 716 coupled to the high-speed interface 708. In other embodiments, multiple processors and / or multiple buses and appropriately multiple memories and multiple types of memories can be used. Additionally, multiple computing devices 700 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0071] The memory 704 stores information within the computing device 700. In one embodiment, the memory 704 is one or more volatile memory units. In another embodiment, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0072] The storage device 706 can provide mass storage for the computing device 700. In one embodiment, the storage device 706 can be or include a computer readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a magnetic tape device, a flash memory or other similar solid-state storage device or array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as a memory 704, a storage device 706, or a memory on a processor 702.

[0073] The high-speed controller 708 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 712 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 710, which can accept various expansion cards (not shown). In this embodiment, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), is coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0074] As shown in the figure, computing device 700 can be implemented in many different forms. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can be implemented in a personal computer such as a laptop computer 722. Alternatively, components from computing device 700 can be combined with other components in a mobile device (not shown) such as device 750. Each of such devices can contain one or more of computing devices 700, 750, and the entire system can be composed of multiple computing devices 700, 750 that communicate with each other.

[0075] Computing device 750 includes processor 752, memory 764, input / output devices such as display 754, communication interface 766 and transceiver 768, among other components. Device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 750, 752, 764, 754, 766 and 768 are interconnected using various buses, and several components may be mounted on a common motherboard or otherwise as appropriate.

[0076] Processor 752 may execute instructions within computing device 750, including instructions stored in memory 764. The processor may be implemented as a chipset including separate or multiple analog and digital processor chips. For example, the processor may provide coordination of other components of device 750, such as control of a user interface, applications executed by device 750, and wireless communications performed by device 750.

[0077] The processor 752 can communicate with the user through a control interface 758 and a display interface 756 coupled to the display 754. The display 754 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display or other appropriate display technology. The display interface 756 may include appropriate circuits for driving the display 754 to present graphics and other information to the user. The control interface 758 can receive commands from the user and convert them for submission to the processor 752. In addition, an external interface 762 in communication with the processor 752 can be provided to enable near-area communication of the device 750 with other devices. The external interface 762 can, for example, be provided for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces can also be used.

[0078] The memory 764 stores information within the computing device 750. The memory 764 may be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An extended memory 774 may also be provided and connected to the device 750 via an extended interface 772, which may include, for example, a SIMM (single in-line memory module) card interface. Such an extended memory 774 may provide additional storage space for the device 750, or may also store applications or other information for the device 750. Specifically, the extended memory 774 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the extended memory 774 may be provided as a security module for the device 750, and may be programmed with instructions that allow the device 750 to be used securely. In addition, secure applications and additional information may be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0079] As described below, the memory may include, for example, flash memory and / or NVRAM memory. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 764, expansion memory 774, or memory on processor 752, which can be received, for example, via transceiver 768 or external interface 762.

[0080] The device 750 can communicate wirelessly through a communication interface 766, which may include digital signal processing circuitry when necessary. The communication interface 766 may provide for communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication may occur, for example, through a radio frequency transceiver 768. In addition, short-range communication may occur, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 770 may provide additional navigation and location-related wireless data to the device 750, which may be used appropriately by applications running on the device 750.

[0081] Device 750 may also communicate audibly using audio codec 760, which may receive spoken information from a user and convert it into usable digital information. Audio codec 760 may likewise generate audible sounds for the user, such as through a speaker, for example, in an earpiece of device 750. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 750.

[0082] As shown in the figure, computing device 750 can be implemented in many different forms. For example, it can be implemented as a cellular phone 780. It can also be implemented as part of a smart phone 782, a personal digital assistant, or other similar mobile device.

[0083] The user can interact with the computing device using a tracked controller 784. In some embodiments, the controller 784 can track the movement of the user's body, such as the movement of the hands, feet, head, and / or torso, and generate input corresponding to the tracked movement. The input can correspond to movement in one or more dimensions of movement, such as in three dimensions. For example, the tracked controller can be a physical controller for a VR application that is associated with one or more virtual controllers in the VR application. As another example, the controller 784 can include a data glove.

[0084] Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to a storage system, at least one input device, and at least one output device.

[0085] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0087] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a Web browser that a user can use to interact with implementations of the systems and techniques described herein), or includes any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0088] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0089] In some embodiments, Figure 7 The computing device depicted in FIG. 7 may include sensors that interface with a virtual reality (VR headset 785). For example, including Figure 7 One or more sensors on the computing device 750 or other computing device depicted in the VR headset 785 may provide input to the VR headset 785, or generally provide input to the VR space. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 750 may use sensors to determine the absolute position of the computing device in the VR space and / or detected rotation, which may then be used as input to the VR space. For example, the computing device 750 may be incorporated into the VR space as a virtual object, such as a controller, a laser pointer, a keyboard, a weapon, etc. Positioning of the computing device / virtual object by the user when incorporated into the VR space may allow the user to position the computing device to view the virtual object in certain ways in the VR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were an actual laser pointer. The user may move the computing device left and right, up and down, in a circle, etc., and use the device in a manner similar to using a laser pointer.

[0090] In some embodiments, one or more input devices included on or connected to the computing device 750 can be used as input to the VR space. The input device can include, but is not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input functionality, a game controller, or other connectable input devices. When the computing device is incorporated into the VR space, user interaction with the input device included on the computing device 750 can cause specific actions to occur in the VR space.

[0091] In some implementations, the touch screen of computing device 750 can be rendered as a touch pad in VR space. A user can interact with the touch screen of computing device 750. For example, in VR headset 785, the interaction is rendered as movement on the rendered touch pad in VR space. The rendered movement can control virtual objects in VR space.

[0092] In some embodiments, one or more output devices included on the computing device 750 may provide output and / or feedback to a user of the VR headset 785 in the VR space. The output and feedback may be visual, tactile, or audio. The output and / or feedback may include, but are not limited to, vibration, turning one or more lights or flashes on and off or blinking and / or flashing, sounding an alarm, playing a ringtone, playing a song, and playing an audio file. The output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), flashes, and speakers.

[0093] In some embodiments, computing device 750 can appear as another object in a computer-generated 3D environment. Interactions by a user with computing device 750 (e.g., rotating, shaking, touching a touch screen, sweeping a finger across a touch screen) can be interpreted as interactions with objects in VR space. In the example of a laser pointer in VR space, computing device 750 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates computing device 750, the user in VR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 750 in VR space on computing device 750 or on VR headset 785.

[0094] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the invention.

[0095] In addition, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve the desired results. In addition, other steps can be provided, or steps can be eliminated from the described flows, and other components can be added to the described systems, or other components can be removed from the described systems. Therefore, other embodiments are within the scope of the following claims.

[0096] Some examples are described below.

[0097] Example 1: A method comprising:

[0098] Receiving, in a first device, a relative description file of physical markers placed at locations, the relative description file defining a relative position of each of the physical markers with respect to at least another of the physical markers;

[0099] Initially locating the position of the first device among the physical markers by visually capturing any first physical marker among the physical markers using an image sensor of the first device; and

[0100] A second physical marker among the physical markers and a location of the second physical marker are identified without line of sight, the second physical marker being identified using the relative description file.

[0101] Example 2: A method according to Example 1, wherein at least the first physical mark includes a code.

[0102] Example 3: A method according to Example 1, wherein at least the first physical mark includes a natural mark.

[0103] Example 4: The method according to at least one of Examples 1 to 3 further includes: presenting an augmented reality environment on a display of the first device, the augmented reality environment including an image captured using at least a camera of the first device and at least a first virtual object applied to the image.

[0104] Example 5: A method according to Example 4, wherein the first virtual object is placed in the image at a predefined position relative to the first physical marker.

[0105] Example 6: A method according to at least one of Examples 1 to 5, wherein the position of the first device is initially located by aligning the camera in a first direction toward the first physical marker, and wherein, after the position of the first device is initially located, a user aligns the camera in a second direction, and wherein identifying the second physical marker includes using the relative description file to determine that the position of the second physical marker is located in the second direction.

[0106] Example 7: The method according to at least one of Examples 4 to 6 further includes: updating the augmented reality environment with a second virtual object at a predefined position relative to the second physical marker.

[0107] Example 8: A method according to Example 6 or 7, wherein the user moves the first device to the location of the second physical marker, and wherein, at the location of the second physical marker, the first physical marker is no longer visible in the augmented reality environment, and the user aims the camera in a third direction toward the location of the first physical marker without line of sight, and the method further includes updating the augmented reality environment with the first virtual object at a predefined position relative to the first physical marker.

[0108] Example 9: A method according to at least one of Examples 6 to 7, wherein the second physical marker is removed from the position of the second physical marker before the second physical marker is identified, and wherein the second physical marker and the position of the second physical marker are identified using the relative description file.

[0109] Example 10: A method according to at least one of the preceding examples, wherein initially locating the position of the first device includes centering a coordinate system of the first device based on the position of the first physical marker, and wherein identifying the second device physical marker does not include re-centering the coordinate system.

[0110] Example 11: The method of Example 10, wherein centering the coordinate system of the first device comprises recalibrating the center of the coordinate system to correct for tracking errors.

[0111] Example 12: A method according to at least one of the preceding examples, wherein initially positioning the first device comprises centering a coordinate system of the first device based on the position of the first physical marker, wherein identifying the second physical marker comprises re-centering the coordinate system, and wherein any virtual objects continue to be presented in the same space via camera pass-through because the relative coordinate scales are the same.

[0112] Example 13: The method according to at least one of the preceding examples further includes providing the relative description file to a second device, wherein the second device is configured to initially locate the position of the second device among the physical markers by visually capturing any one of the physical markers using an image sensor of the second device, and use the relative description file to identify the positions of all remaining physical markers among the physical markers without line of sight.

[0113] Example 14: A method according to at least one of the preceding examples, wherein the relative description file is a first relative description file, the method further comprising receiving a second relative description file from a second device, and at least partially merging the first relative description file and the second relative description file.

[0114] Example 15: A method according to at least one of the preceding examples, wherein the relative description file defines a relative position of the second physical mark with respect to the first physical mark, and the relative position includes a displacement of the second physical mark relative to the first physical mark.

[0115] Example 16: A method according to at least one of the preceding examples, wherein the first device identifies at least some of the physical tags, and the method further includes generating a log on the first device reflecting the at least some of the physical tags that the first device has identified.

[0116] Example 17: A method according to at least one of the preceding examples, wherein the relative description file defines the relative position of each of the physical tags with respect to all other physical tags in the physical tags.

[0117] Example 18: A non-transitory storage medium having stored thereon instructions, the instructions being configured, when executed, to cause a processor to perform operations comprising:

[0118] Receiving, in a first device, a relative description file of physical markers placed at locations, the relative description file defining a relative position of each of the physical markers with respect to at least another of the physical markers;

[0119] Initially locating the position of the first device among the physical markers by visually capturing any first physical marker among the physical markers using an image sensor of the first device; and

[0120] A second physical marker among the physical markers and a location of the second physical marker are identified without line of sight, the second physical marker being identified using the relative description file.

[0121] Example 19: A system comprising:

[0122] A first device, the first device being configured to receive, for each of the physical markers placed at the location, a location point determined using the first device, each of the location points being associated with a corresponding one of the physical markers, the first device being further configured to generate a relative description file of the physical markers based on the location point, the relative description file defining a relative position of each of the physical markers with respect to at least another one of the physical markers; and

[0123] A second device and a third device, each of which is configured to receive the relative description file generated by the first device, wherein each of the second device and the third device is further configured to initially locate the position of the second device or the third device among the physical markers by visually capturing any one of the physical markers using an image sensor of the second device or the third device, respectively, and identify the positions of all remaining physical markers among the physical markers using the relative description file without line of sight.

[0124] Example 20: A method comprising:

[0125] receiving, in a first device, for each of the physical markers positioned at the location, a location point determined using the first device, each of the location points being associated with a respective one of the physical markers;

[0126] generating a relative description file of the physical markers based on the position points, the relative description file including a relative position of each of the physical markers with respect to at least another of the physical markers; and

[0127] The relative description file is provided to at least a second device and a third device, wherein each of the second device and the third device is configured to initially locate the position of the second device or the third device among the physical markers by visually capturing any one of the physical markers using an image sensor of the second device or the third device, respectively, and to identify the positions of all remaining physical markers among the physical markers using the relative description file without line of sight.

[0128] Example 21: A method according to Example 20, wherein at least one of the physical markers includes a code.

[0129] Example 22: A method according to Example 20, wherein at least one of the physical markers includes a natural marker.

Claims

1. A method for identifying a physical marker, include: presenting an augmented reality environment on a display of a device, the augmented reality environment comprising at least an image captured using an image sensor of the device and at least a virtual object applied to the image; locating the device among the physical markers by aligning the image sensor in a first direction toward a first physical marker from the physical markers, the physical markers having relative positions defined in a relative description file; identifying the first physical marker using the image sensor; After the position of the device is located, aligning the image sensor in a second direction toward a second one of the physical marks; identifying the second physical marker and the location of the second physical marker without requiring a line of sight between the image sensor and the second physical marker, the second physical marker being identified using the relative description file; and updating the augmented reality environment with the virtual object at a predefined location relative to the first physical marker in response to: The device is moved to a location of the second physical marker, at which the first physical marker is no longer visible in the augmented reality environment, and The image sensor is aligned in a third direction toward the location of the first physical marker without requiring a line of sight between the image sensor and the first physical marker.

2. The method according to claim 1, in, At least the first physical marking comprises a code.

3. The method according to claim 1, in, At least the first physical marking comprises a natural marking.

4. The method according to claim 1, in, The virtual object is positioned in the image at a predefined position relative to the first physical marker.

5. The method according to any one of claims 1 to 4, in, After the position of the device is located, identifying the second physical mark includes using the relative description file to determine that the position of the second physical mark is located in the second direction.

6. The method according to claim 5, in, The virtual object is a first virtual object, The method further includes updating the augmented reality environment with a second virtual object at a predefined position relative to the second physical marker.

7. The method according to claim 5, in, Before the second physical marker is identified, the second physical marker is removed from the location of the second physical marker, and wherein the second physical marker and the location of the second physical marker are identified using the relative description file.

8. The method according to claim 1, in, Locating the position of the device includes centering a coordinate system of the device based on the position of the first physical marker, and wherein identifying the second physical marker does not include re-centering the coordinate system.

9. The method according to claim 8, in, Centering the coordinate system of the device includes recalibrating the center of the coordinate system to correct for tracking errors.

10. The method according to claim 1, in, Locating the position of the device includes centering the coordinate system of the device based on the position of the first physical marker, wherein identifying the second physical marker includes re-centering the coordinate system, and wherein any virtual objects continue to be presented in the same space via camera pass-through because the relative coordinate scale is the same.

11. The method according to claim 1, in, the device being a first device, The method further includes providing the relative description file to a second device, wherein the second device is configured to initially locate the second device among the physical markers by visually capturing any one of the physical markers using an image sensor of the second device, and to use the relative description file to identify the positions of remaining physical markers among the physical markers without line of sight.

12. The method according to claim 1, in, The device is a first device, the relative description file is a first relative description file, and the method further includes receiving a second relative description file from a second device, and at least partially merging the first relative description file and the second relative description file.

13. The method according to claim 1, in, The relative description file defines the relative position of the second physical mark with respect to the first physical mark, and the relative position includes the displacement of the second physical mark relative to the first physical mark.

14. The method according to claim 1, in, The device recognizes at least some of the physical tags, and the method further includes generating a log on the device reflecting the at least some of the physical tags that the device has recognized.

15. The method according to claim 1, in, The relative description file defines the relative position of each of the physical tags with respect to other physical tags in the physical tags.

16. A non-transitory storage medium having stored thereon instructions, the instructions being configured to cause a processor to perform operations when executed, the operations include: presenting an augmented reality environment on a display of a device, the augmented reality environment comprising at least an image captured using an image sensor of the device and at least a virtual object applied to the image; locating the device among the physical markers by aligning the image sensor in a first direction toward a first physical marker from the physical markers, the physical markers having relative positions defined in a relative description file; identifying the first physical marker using the image sensor; After the position of the device is located, aligning the image sensor in a second direction toward a second one of the physical marks; identifying the second physical marker and the location of the second physical marker without requiring a line of sight between the image sensor and the second physical marker, the second physical marker being identified using the relative description file; and updating the augmented reality environment with the virtual object at a predefined location relative to the first physical marker in response to: The device is moved to a location of the second physical marker, at which the first physical marker is no longer visible in the augmented reality environment, and The image sensor is aligned in a third direction toward the location of the first physical marker without requiring a line of sight between the image sensor and the first physical marker.

17. A device for identifying a physical marker, include: at least one processor; as well as A non-transitory storage medium on which is stored: a first relative description file of the physical tags placed at the locations, the first relative description file defining a relative location of each of the physical tags with respect to at least another of the physical tags; as well as instructions, which when executed are configured to cause the at least one processor to perform operations comprising: presenting an augmented reality environment on a display of a device, the augmented reality environment comprising at least an image captured using an image sensor of the device and at least a virtual object applied to the image; locating the device among the physical markers by aligning the image sensor in a first direction toward a first physical marker from the physical markers, the physical marker having a relative position defined in the first relative description file; identifying the first physical marker using the image sensor; After the position of the device is located, aligning the image sensor in a second direction toward a second one of the physical marks; identifying the second physical marker and the location of the second physical marker without requiring a line of sight between the image sensor and the second physical marker, the second physical marker being identified using the first relative description file; and updating the augmented reality environment with the virtual object at a predefined location relative to the first physical marker in response to: The device is moved to a location of the second physical marker, at which the first physical marker is no longer visible in the augmented reality environment, and The image sensor is aligned in a third direction toward the location of the first physical marker without requiring a line of sight between the image sensor and the first physical marker.

18. The device according to claim 17, in, The first relative description file defines a first relative position connecting the first physical mark and the second physical mark, and wherein the first relative description file does not define a second relative position connecting the second physical mark and a third physical mark.

19. The device according to claim 18, in, The operations further include receiving a second relative description file defining the second relative position, and merging, at least in part, the first relative description file and the second relative description file into a third relative description file reflecting the first relative position and the second relative position.

Citation Information

Patent Citations

  • Positional recognition for augmented reality environment

    US20190156577A1

  • System for evaluating identification marks, identification marks and use thereof

    CN103649775A

  • Facilitating dynamic non-visual markers for augmented reality on computing devices

    CN107111361A

  • Schemes for retrieving and associating content items with real-world objects using augmented reality and object recognition

    CN107111740A

  • Image processing device, image processing method, program, and terminal device

    US20150302645A1