COMPUTER-IMPLEMENTED METHOD FOR POSITIONING AN ELECTRONIC DEVICE IN A COORDINATE SYSTEM - Patent application

Abstract

The present invention relates to a computer-implemented method for determining a position and orientation of an electronic device in a coordinate system, the method comprising: capturing an image of a reference object by a camera of the electronic device, extracting an address for accessing a domain from the image of the reference object, the domain containing information about the position and orientation and known physical dimensions and appearance of the reference object in the coordinate system, obtaining the position and orientation in the coordinate system and the known physical dimensions and appearance by accessing the domain for the reference object, determining a relative distance and orientation of the electronic device with respect to the reference object based on the known physical dimensions and appearance of the reference object and the apparent dimensions and appearance of the reference object captured in the captured image, and determining a position and orientation of the electronic device in the coordinate system based on the position and orientation of the reference object in the coordinate system and the determined relative distance and orientation of the electronic device with respect to the reference object.

Description

[Technical field]

[0001] The present invention relates to the field of positioning, and more particularly to a method and system for determining the position and orientation of an electronic device within a coordinate system. [Background technology]

[0002] Many applications in industry and entertainment, such as mapping, warehouse management, and gaming, require rapid and accurate positioning of camera-enabled devices such as smartphones in a coordinate system that can be either local or global. GPS (Global Positioning System) has limited accuracy, even less so indoors, and often offers unsatisfactory or incomplete solutions to this problem. Many applications require accuracy one to two orders of magnitude higher than GPS, for example down to the centimeter or millimeter scale, and need to work indoors as well as outdoors. Furthermore, many applications require not only accurate calculation of the (relative) three-dimensional position, but also accurate calculation of the (relative) three-dimensional orientation of the device in space, sometimes with similarly high accuracy. Also, many applications can make do with local positioning and do not inherently involve global positioning.

[0003] Yet another type of application does not essentially involve positioning devices with respect to the local environment, but rather with positioning devices relative to one another. Such applications may include joint augmented reality (AR) experiences or robotics. In the case of AR in particular, improvements are needed to enable two camera-enabled devices that were previously unaware of one another to quickly and precisely determine their position and orientation within a shared space, or to nonetheless quickly and precisely determine their relative position and orientation.

[0004] However, current technology has limitations when it comes to sharing virtual worlds or partial virtual worlds between users. In the case of augmented reality in particular, improvements are needed to enable camera-enabled devices to quickly, precisely, and safely determine their relative positions and orientations within a shared space, and within a local, physically determined coordinate system in which zero or more augmented reality assets can be embedded. Summary of the Invention

[0005] In view of the above, it is an object of the present invention to provide a method for locating an electronic device in a coordinate system, which may in particular be a method for locating a camera-enabled device in a coordinate system via a centralized domain (e.g. a remote server).

[0006] Specifically, the inventors of the inventive concept have realized a method that utilizes real-world reference objects as a way to anchor the virtual world to the real world, and also to transmit the position and orientation back to the device as needed, together with a cloud-based system, to achieve an effective, secure, and accurate way of tracking the relative and absolute position and orientation of an electronic device navigating a virtual world, relative to the virtual world's own local coordinate system or a shared global coordinate system. In other words, this method allows a camera-enabled device to utilize a centralized domain to position itself in a local coordinate system, which does not need to be defined relative to a fixed landmark, but can instead be determined by another electronic device.

[0007] The advantages of this method are that it is geometrically agnostic, in the sense that the two devices do not need to have prior knowledge of each other's geometry and do not need to establish an electronic handshake at run-time, and that it is privacy-preserving. Also, a first device allows a second device to learn the location and orientation of the second device in a local coordinate system (more specifically, any coordinate system in which the location and orientation of the first device are already known), which may be possible without requiring the two devices to be a priori familiar with each other and without requiring the two devices to establish an electronic handshake or exchange messages in the field.

[0008] The method is fast, works indoors as well as outdoors, provides high position accuracy compared to, for example, GPS, and offers the possibility of deployment and maintenance via crowdsourcing.

[0009] According to a first aspect, a computer-implemented method for determining a position and orientation of an electronic device in a coordinate system is provided, the method includes: capturing an image of a reference object by a camera of the electronic device; extracting from the image of the reference object an address for accessing a domain, the domain including information on a position and orientation of one or more reference objects in the coordinate system and known physical dimensions and appearances of the one or more reference objects; obtaining a position and orientation in the coordinate system and the known physical dimensions and appearances of the reference objects by accessing the domain; determining a relative distance and orientation of the electronic device to the reference object based on the known physical dimensions and appearances of the reference object and the apparent dimensions and appearances of the reference object captured in the captured image; and determining a position and orientation of the electronic device in the coordinate system based on the position and orientation of the reference object in the coordinate system and the determined relative distance and orientation of the electronic device to the reference object.

[0010] The method may further comprise positioning the electronic device at a position and orientation in the coordinate system corresponding to the determined position and orientation of the electronic device in the coordinate system. Thus, the method may be a method for positioning the electronic device in a coordinate system.

[0011] In the above, the term "coordinate system" can be interpreted as a set of right-handed Cartesian axes positioned and oriented in a particular fixed relationship to a physical object. A coordinate system may have some association with a given surrounding physical region or area, or may have no such a priori association. For example, a coordinate system may be attached to a building, with the building itself being the associated region, or the neighborhood containing the building, or the entire town, city, or larger geographic area may be the associated region. A coordinate system may also be attached to a movable physical object. For example, a coordinate system may be attached to a cargo container, with the interior of the container being the associated region, or a coordinate system may be attached to a camera of an electronic device, with no association to an a priori region around the device. A coordinate system may be a global coordinate system or a local coordinate system. A coordinate system may be a sub-coordinate system to another coordinate system. The expression "locating", as in "locating an electronic device in a coordinate system", refers herein to determining a particular position and orientation in which the electronic device must be located within the coordinate system, and placing the electronic device at the determined position and orientation. In other words, positioning the electronic device can be thought of as establishing the electronic device in a determined position and orientation within a coordinate system. In other words, it can be thought of as calibrating the electronic device with respect to the coordinate system. Thus, this method allows the electronic device to position itself within such coordinate system, as described above, by using reference objects positioned within the environment covered by the coordinate system.

[0012] The term "position" can be interpreted as determining the position and orientation of one coordinate system relative to another. Moreover, it can mean determining the position and orientation of a physical object in the sense of positioning a default coordinate system attached to a physical object in one coordinate system relative to the former coordinate system. In particular, as will be appreciated by those skilled in the art, a coordinate system is positioned relative to another coordinate system only if there exists a concisely encodable and efficiently computable function that transforms a point in Cartesian coordinates in a space represented in one coordinate system to the Cartesian coordinates of a point in a space represented in the other coordinate system. This function or its encoding can be called a "coordinate transformation". Thus, an electronic device can be positioned in a coordinate system by determining the position and orientation of the local coordinate system of the electronic device relative to the coordinate system in which the electronic device is to be positioned.

[0013] The term "address" shall be interpreted as a link or pointer indicating where to access the domain. A domain shall be understood as an online database or centralized hub, accessed via a private or public API. The database may be distributed. In some cases, subject to spatial constraints, pieces of the database may be directly replicated in the code of the reference object. The latter may be particularly easily performed, for example, when the reference object is generated by an online electronic device that communicates with the domain and can edit its appearance on the fly. A domain may be associated with a default local coordinate system, called the "domain's default coordinate system". The domain may host / manage the coordinate system. The domain may be a remote server, for example a cloud implementation server. The domain may further host or manage a virtual space or an AR world in the coordinate system. The domain may contain information related to the virtual objects shown in the virtual space. The reference objects shall be interpreted as physical real-world objects. Each reference object has known physical dimensions and appearance in the real world. More specifically, the reference object may be located in the environment covered by the coordinate system. Furthermore, each reference object may have a known position and orientation in the real world. The known position and orientation of the reference object may be GPS coordinates and orientation relative to the real world. Alternatively, the known position and orientation of the reference object may be known relative to other real world objects. For example, it may be a position and orientation relative to a building, room, or other reference object. Furthermore, each reference object is associated with a position and orientation in the coordinate system. The reference object may thus act as a fixturing device between the real world and the coordinate system. The reference object may be a fiducial marker. The reference object may be a machine-readable optical tag, such as a QR code or a bar code, printed or displayed at a fixed position on a rigid surface, or may be made of any material with a non-malleable geometry.The surface may be the screen or display of an electronic device. The reference object may be assumed to have a commonly recognizable geometric shape that clearly distinguishes the "outside view" from the "inside view" of the reference object when it is displayed on an opaque surface with a well-defined outward normal, or when it is printed on glass. It may also be assumed that a unique coordinate system can be associated with each reference object by using a default set of conventions that depend only on the general type of reference object. For example, for a QR code printed on a flat surface, in a typical "portrait" view of the QR code, the axes of the coordinate system are positioned at the top left of the QR code, and the axis direction is determined by the plane containing the QR code and the orientation of the QR code in that plane. By "the coordinate system of a reference object" or "the reference object's coordinate system" we mean the latter unique coordinate system associated with the reference object, determined by the selected convention.

[0014] The optical tag may contain digital content. The digital content may be referred to as the "code" of the reference object. The code of the reference object may include any subset of: (i) the id of the "reference object type", (ii) the unique id of the reference object, (iii) the domain address, (iv) any metadata, such as a piece of data stored on the domain, or other human readable or arbitrary data (subject to spatial constraints). In the case of a reference object printed on a digital display or an electronic ink display, the third and fourth types of content may change over time, but the first or second types are assumed not to change over time (or these two reference objects appearing at different times are considered as separate).

[0015] The reference object type id may identify what type of reference object it is. The reference object unique id, or simply the reference object identification, may be a unique identifier of the reference object.

[0016] The physical dimensions and appearance of the reference object may include any dimensional data of the reference object. This means any type of data encoded in some agreed format that allows recovering the true physical dimensions and appearance or shape of the reference object from a purely two-dimensional image of the reference object captured by an imaging device at a common angle, without any other context or image. For example, if the reference object is a QR code printed on a flat surface, the dimensional data may be the length of the sides of the QR code measured in meters (e.g. a single number). If the reference object is a barcode printed on a flat surface, the dimensional data may be either the height or width of the barcode, or both. If the reference object is a QR code printed on a cylindrical surface, such data may include the same information (i.e. the presence of a cylindrical surface), the radius of the cylinder, the length of the sides of the QR code, and the angle of the normal axis of the QR code with respect to the main axis of the cylinder, etc. Thus, the variety and complexity of the dimensional data should be broadly understood as a function of the variety of surfaces used as substrates for the reference objects. It can be assumed that two reference objects with the same reference object type id may share the same dimensional data. In particular, the central domain can associate dimensional data of a reference object with an electronic device by providing only the id of the reference object's reference object type.

[0017] "Image of a reference object" or "appearance of a reference object" means a rasterized two-dimensional image or a series of images of a reference object captured by a camera of an electronic device from a common angle and distance, which allows, for example, the device to parse the machine-readable content of the reference object.

[0018] As will be appreciated by those skilled in the art, the image of the reference object is used in conjunction with dimensional data of the reference object to determine the position of the device's camera relative to the reference object's coordinate system. More specifically, these two pieces of information can jointly determine a coordinate transformation from the reference object's coordinate system to a coordinate system attached to the device's camera. This coordinate transformation can then determine the device's position within the reference object's coordinate system.

[0019] An electronic device can be said to "scan" a reference object when it obtains the code of the reference object via an image of the camera of the electronic device itself. By scanning the reference object, the device thus obtains the id of the reference object's reference object type, if one exists.

[0020] In a given system, the domain API may include functions to obtain the dimensional data corresponding to a reference object type id given the id of the reference object type, and may also include functions to obtain the position and orientation of a reference object within the default coordinate system of the domain by providing the unique id of the reference object.

[0021] Thus, a device scanning a reference object knows the dimensional data (i.e., known physical dimensions and appearance) of the reference object, either through a priori or a posteriori (i.e., by scanning) knowledge of the domain address, and possibly by making an API call to the domain address. The same API also knows the position and orientation of the reference object in the domain's default coordinate system. By mathematically composing the two positions (i.e., a coordinate transformation), the device knows its own position and orientation in the domain's default coordinate system.

[0022] The term "known," as in "known physical dimensions and appearance," means that the physical dimensions and appearance are predetermined. In other words, the reference object has physical dimensions and appearance that are known in advance and can be compared to the apparent dimensions and appearance of the imaged reference object.

[0023] The term "apparent," as in "apparent size and appearance," refers to the size and appearance as seen when imaged by a device's camera or other sensor (such as LIDAR). In other words, the apparent size and appearance of a reference object may differ from its known physical size and appearance.

[0024] The orientation of the electronic device (or a reference object) may include information about how the electronic device (or a reference object) is rotated relative to a coordinate system.

[0025] The known appearance may include, for example, the shape and / or pattern of the reference object. The known appearance may include information about the orientation of the reference object. As an example, the appearance may include information about how the reference object is rotating.

[0026] As will be appreciated by those skilled in the art, when an object having known physical dimensions and appearance is imaged by a device, the relative distance and orientation of the device to the object can be determined by comparing the object's known physical dimensions and appearance with the object's apparent dimensions and appearance. This can be easily and accurately achieved by providing a reference object as a well-defined feature, i.e. an object having well-defined dimensions and appearance (e.g., as a two-dimensional object with clear boundaries).

[0027] A possible advantage of the proposed method is the high accuracy of optical positioning compared to e.g. GPS positioning. Another possible advantage applies to indoor use where GPS positioning is even less accurate. Another feature of the method is that it does not require a peer-to-peer communication session, but only a communication session between the device and the domain, thus improving privacy on the one hand and increasing usability and scalability on the other hand, since each device only needs to communicate with one type of entity (i.e. the domain) instead of several different types of entities (various other devices).

[0028] Further important use cases and advantages of this method can be found as they arise from extensions and modifications that are naturally implemented, as described below, including, as a non-limiting use case described below, specifically, that two or more previously unknown devices can enter into a highly accurate shared augmented reality (AR) experience without the need for an electronic handshake and without access to existing nearby reference objects.

[0029] Furthermore, a reference object can serve as a reference object for several different coordinate systems at once: for example, a reference object located at a shopping mall can serve as a reference object for the coordinate system of the shopping mall and for the coordinate system of the city in which the shopping mall is located.

[0030] Finally, domains allow for a simple, resilient and flexible scheme for expanding the network of reference objects, since the information stored in the domain can be easily updated - in other words, reference objects can be added and / or removed by updating the information stored in the domain.

[0031] In a non-limiting example, the proposed method allows positioning an electronic device in an augmented reality world using a coordinate system.

[0032] The method may further include connecting the electronic device to the virtual space defined by the coordinate system at the determined position and orientation within the coordinate system. Connecting the electronic device to the virtual space enables the device to view and interact with the virtual space.

[0033] The virtual space may be a partially virtual or mixed real world, such as an augmented reality world, i.e. the virtual space may be a virtual part of an augmented reality world. However, it should be noted that the partial or mixed real world experienced by the user may be entirely virtual from a visual point of view, but still anchored in the real world. For example, real world objects may be superimposed on virtual objects. The virtual space may be an existing virtual space, i.e. the virtual space may exist prior to connecting the electronic device to the virtual space.

[0034] A virtual space "defined by" a coordinate system may be interpreted as a virtual space that exists within or is fixed to a coordinate system. In other words, the virtual space may utilize the coordinate system as a frame of reference. The virtual space may have its own relative coordinate system, which may be a sub-coordinate system of a larger coordinate system, as described above. Connecting an electronic device to a virtual space may be thought of as positioning the electronic device in the relative coordinate system of the virtual space.

[0035] The domain may host multiple virtual spaces. Each virtual space of the multiple virtual spaces may be defined by a coordinate system. In other words, the coordinate system may be a coordinate system common to the multiple virtual spaces. The method may further include determining which of the multiple virtual spaces to connect the electronic device to.

[0036] The ability of a domain to host multiple virtual spaces allows for greater use of the same coordinate system. Additionally, having multiple virtual spaces defined by the same coordinate system and hosted by the same domain facilitates a simplified process for an electronic device (or a user of the electronic device) to connect to a virtual space of its choice.

[0037] The method may further include extracting a reference object identification from the image of the reference object. Obtaining the position and orientation, and the known physical dimensions and appearance may include transmitting the reference object identification to the domain. The reference object identification can be used to access the correct characteristics in the domain corresponding to the reference object (i.e., the position and orientation, and the known physical dimensions and appearance of the reference object).

[0038] The domain may include a database having entries for a plurality of reference objects, each entry including a reference object identification, a position and orientation in a coordinate system of the corresponding reference object, and known physical dimensions and appearance of the corresponding reference object. Obtaining the position and orientation, and the known physical dimensions and appearance may further include accessing an entry in the database corresponding to the reference object identification.

[0039] According to a second aspect, there is provided a system for enabling locating an electronic device within a coordinate system. The system comprises a plurality of reference objects, each reference object having a reference object identification and known physical dimensions and appearance, and a domain hosting the coordinate system. The domain includes a database having entries for the plurality of reference objects with associated addresses. Each entry includes a reference object identification, a position and orientation within the coordinate system of the corresponding reference object, and the known physical dimensions and appearance of the corresponding reference object. Each reference object further includes an address relative to the domain.

[0040] The system makes it possible to position an electronic device in a coordinate system according to the above-described method according to the first aspect.

[0041] A possible advantage with the approach of being able to locate electronic devices within a coordinate system is that multiple electronic devices can be located within the coordinate system without the devices having to communicate with each other. This can be advantageous in that the process of locating and / or connecting electronic devices is simplified as the electronic devices do not have to be connected to each other. For the same reason, the process becomes more secure.

[0042] The multiple reference objects can be reference objects located at different locations in the real world. By distributing the multiple reference objects in the real world, the electronic device can connect to a coordinate system by capturing an image of the reference object wherever the reference object is located.

[0043] The reference objects may be non-transient or persistent.

[0044] The reference objects may be machine-readable optical labels, which may be, for example, two-dimensional codes such as QR codes.

[0045] A domain can host multiple virtual spaces, all defined by coordinate systems. One or more of the multiple virtual spaces can be associated with a reference object of the multiple reference objects.

[0046] Domains can be configured to track multiple electronic devices in each virtual space.

[0047] The term "tracking" refers to tracking the position and orientation of an electronic device as it moves around. Tracking multiple electronic devices connected to a virtual space can be performed by receiving movement data from the multiple electronic devices. The movement data can be obtained, for example, from an accelerometer and a gyroscope of each electronic device.

[0048] A domain may be configured to track multiple coordinate systems, including coordinate transformations between pairs of multiple coordinate systems.

[0049] The system may further include a temporary reference object displayed on a display of the electronic device positioned within the coordinate system. The temporary reference object may have a reference object identification, known physical dimensions and appearance, and an address for the domain. The database may further include an entry for the temporary reference object. The entry may include a reference object identification of the temporary reference object displayed on the display of the electronic device, a position and orientation of the temporary reference object within the coordinate system, and the known physical dimensions and appearance of the temporary reference object. The entry for the temporary reference object may further include a virtual space identification that identifies the virtual space to which the device is connected.

[0050] A possible related advantage is that if a non-temporary reference object is not available, an electronic device already positioned within the coordinate system can instead display a temporary reference object to enable an electronic device that is to be positioned within the coordinate system to capture an image of the temporary reference object.

[0051] The above-mentioned features of the first aspect also apply to the second aspect, where applicable, and in order to avoid excessive repetition, reference is made to the above.

[0052] According to a third aspect, there is provided a non-transitory computer readable recording medium having program code portions recorded thereon, the program code portions performing a method according to the first aspect when executed on a device having processing capability.

[0053] The program code portions may be applications that can be downloaded to the electronic device via an application that provides the service. Some of the program code portions are executable in a domain.

[0054] The above-mentioned features of the first and second aspects also apply to the third aspect, where applicable, and in order to avoid excessive repetition, reference is made to the above.

[0055] Further objects, features, aspects, and advantages of the present invention will become apparent from the following detailed description and drawings. Unless otherwise specified, the same features and advantages described with respect to one embodiment are applicable to other embodiments. [Brief description of the drawings]

[0056] These and other aspects of the invention will now be described in more detail with reference to the accompanying drawings, which show variants of the invention, and which should not be construed as limiting the invention to the specific variants, but instead are used to explain and understand the concepts of the invention.

[0057] As shown in the drawings, the sizes of layers and regions have been exaggerated for purposes of explanation and are thus provided to illustrate the general structure of the variations of the inventive concepts. Like reference numbers represent like elements throughout the drawings.

[0058] [Figure 1] 1 illustrates diagrammatically a system for enabling positioning of an electronic device within a coordinate system; [Diagram 2] 1 is a flowchart illustrating steps of a computer-implemented method for positioning an electronic device within a coordinate system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0059] The inventive concept will now be more fully described with reference to the accompanying drawings, in which presently preferred versions of the invention are shown. However, the inventive concept may be embodied in many different forms and should not be construed as being limited to the versions set forth herein. Rather, these versions are provided for completeness and completeness, to fully convey the scope of the inventive concept to those skilled in the art.

[0060] 1 and 2, a computer implemented method for locating an electronic device within a coordinate system, and a system for enabling locating an electronic device within a coordinate system, will now be described.

[0061] 1 shows a schematic of a system 100 for enabling positioning of an electronic device 110 within a coordinate system. It should be noted that the system 100 enables positioning of multiple electronic devices within the same coordinate system. Furthermore, the system can simultaneously host multiple virtual spaces, e.g. augmented reality worlds, defined by coordinate systems.

[0062] The system 100 includes a number of reference objects 102. The number of reference objects 102 may be machine-readable optical labels. Thus, the reference objects 102 can be read by an optical sensor of the electronic device 110, such as a camera 112 or a LIDAR sensor. The number of reference objects 102 may be, for example, a QR code or a barcode. Information encoded in the machine-readable optical label can be obtained by decoding the machine-readable optical label.

[0063] The multiple reference objects 102 may be located at various locations in the real world. As an example, the multiple reference objects 102 may be distributed in indoor or outdoor spaces, such as a train station, a city center, a park, a museum, a hotel, etc.

[0064] One or more of the reference objects 102 may be non-transient. In other words, the reference objects 102 may be permanent. In FIG. 1, the system is illustrated as having two non-transient reference objects 102, represented by two rectangles shown separately from the electronic device. However, the system may include any number of reference objects 102. The reference objects 102 may be non-transient in the sense that they have a static position and orientation in the virtual space. Furthermore, the non-transient reference objects 102 may be fixedly positioned on various surfaces. For example, the non-transient reference objects 102 may be positioned on a physical wall, object, or other surface in the real world. The reference objects 102 may be permanent in the sense that they can be printed and are not changeable. Alternatively, the reference objects may be displayed on a changeable panel, such as an electronic paper display.

[0065] Alternatively, or in combination with the above, one or more of the reference objects 102 may be temporary reference objects 102. The reference objects 102 may be temporary in the sense that they have a dynamic position and orientation in the virtual space (and in the real world), i.e., their position and orientation may change. Furthermore, the reference objects 102 may be temporary in that they may be generated temporarily. As an example, the temporary reference object 102 may be a reference object displayed on the display 116 of an electronic device that is already located in the virtual space 114, as shown here by the electronic device 114 on the right side of FIG. 1. Thus, the temporary reference object 102 may serve the purpose of allowing the electronic device 110 to be located in the virtual space to utilize an electronic device already located in the virtual space 114, for example, if a non-temporary reference object 102 is not available.

[0066] A temporary or mutable non-transient reference object can be updated to store information associated with each reference object that would otherwise be stored in a domain (described further below). This allows the reference object to be used offline, i.e., independent of the domain. This can be advantageous, for example, when the domain is not accessible.

[0067] Each reference object 102 of a plurality of reference objects (both transient and non-transient) may have an associated reference object identification. The reference object identification may be used as a unique identifier of the corresponding reference object. The reference object identification may be encoded in a machine-readable optical label that may configure the reference object 102. The reference object identification may be a unique numeric, alphabetic, or alphanumeric string. For example, the reference object identification may be an HTML URL code.

[0068] Each reference object 102 of the multiple reference objects may have associated known physical dimensions and appearance. The known physical dimensions and appearance of the reference object may be encoded in a machine-readable optical label that may configure the reference object 102. Thus, the known physical dimensions and appearance of the reference object may be obtained directly from the reference object without the need to communicate with the domain (described further below). The term "known", as in "known physical dimensions and appearance", means that the physical dimensions and appearance are predetermined. In other words, the reference object 102 has physical dimensions and appearances that are known in advance and can be compared to the apparent dimensions and appearances of the reference object captured in the captured image of the reference object 102. The appearance may include, for example, the shape and / or pattern of the reference object 102. The appearance may include information regarding the orientation of the reference object 102. As an example, the appearance may include information regarding how the reference object 102 is rotating.

[0069] Each reference object 102 of the multiple reference objects may have a position and orientation in an associated coordinate system. A reference object 102 that is non-transient may be assigned a position and orientation during a calibration process that is performed before the reference object is made available to the electronic device 110. A reference object 102 that is transient may have a position and orientation in a coordinate system that is continuously updated as it moves around in the real world. The position and orientation of the reference object in the coordinate system may be encoded in a machine-readable optical label that may configure the reference object 102. Thus, the position and orientation of the reference object 102 may be extracted from an image of the reference object.

[0070] Multiple reference objects can point to the same domain and provide different "entrances" for electronic devices to align themselves to the default coordinate system of the domain. In this case, each reference object can specify a unique reference object id as described above, or specify such a unique id only implicitly, but computable according to some agreed mechanic of the scheme from some other visual characteristic of the reference object, such as color. However, it is also possible to completely avoid reference object ids even in the presence of multiple reference objects, for example by encoding useful information (e.g. the position of the reference object in the default coordinate system) directly in the content of the reference object's code.

[0071] In a non-limiting example, in a location with a small number of reference objects, such as a home office or playroom, generating reference object ids by color coding may be useful.

[0072] The electronic device 110 to be located in the coordinate system (and electronic devices already located in the coordinate system 114) is shown here with dashed lines to indicate that they may not form part of the system 100. However, it should be noted that the electronic device that may be located in the coordinate system may be any suitable electronic device, not just a mobile phone as shown here. For example, the electronic device 110 may be a smartphone, a tablet, a laptop, a smart watch, smart glasses, AR glasses, AR lenses, or any other suitable device. The electronic device 110 may communicate with the domain (described further below) via a wireless connection, such as a Bluetooth connection, a WiFi network, or a mobile network (e.g., 3G, 4G, 5G, etc.).

[0073] The system 100 further includes a domain 104, which can be interpreted as a remote server or a remote location in the cloud that manages and controls the system. The domain 104 can act as a centralized hub for the system 100 in that it communicates with multiple reference objects 102 and any electronic devices 110 positioned within the coordinate system, as indicated by the arrows in FIG. 1.

[0074] The domain 104 may host one or more virtual spaces. The one or more virtual spaces may be defined by a coordinate system as described above. Each virtual space may have its own relative coordinate system. Each relative coordinate system may have a well-defined relationship to the coordinate system. Each virtual space may host multiple interacting electronic devices. Each virtual space may have an associated sub-address to allow an electronic device that is to be located in the coordinate system (or virtual space) 110 to specify a particular virtual space. The domain may be configured to track the electronic devices located within the coordinate system. Furthermore, the domain may track, in each virtual space it hosts, one or more electronic devices connected to each virtual space. Tracking of the electronic devices may be performed by receiving movement data from the multiple electronic devices. Furthermore, the domain may update the position and / or orientation of the electronic devices connected to the coordinate system (or virtual space) based on the received movement data. The movement data may be obtained by sensors of each electronic device, such as an accelerometer or a gyroscope. The position and orientation of the electronic device may be defined relative to a local coordinate system of the electronic device having an origin at the origin of the camera of the electronic device.

[0075] The domain 104 may include a server 106 and a database 108 for controlling and operating the system. The server 106 may include circuitry. The circuitry may physically include a single circuit device. Alternatively, the circuitry may be distributed across several circuit devices. The server 106 may further include a transceiver. The circuitry may be communicatively coupled to the transceiver and memory. The circuitry may include a data bus, and the circuitry may communicate with the transceiver and / or memory via the data bus.

[0076] The circuitry may be configured to perform overall control of the functions and operations of the server 106. The circuitry may include a processor. The processor may be, for example, a central processing unit (CPU), a microcontroller, or a microprocessor. The processor may be configured to execute program code stored in the memory to perform the functions and operations of the server 106.

[0077] The transceiver may be configured to enable the server 106 to communicate with other devices, such as the electronic device 110. The transceiver may be capable of both transmitting data to the server 106 and receiving data from the server 106.

[0078] The memory may be a non-transitory computer-readable storage medium. The memory may include one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), or other suitable devices. In a typical configuration, the memory may include a non-volatile memory for long-term data storage and a volatile memory that serves as the system memory of the server 106. The memory may exchange data with the circuitry via a data bus. There may also be associated control lines and address buses between the memory and the circuitry.

[0079] The functions and operations of the server 106 can be implemented in the form of executable logic routines (e.g. lines of code, software programs, etc.) stored in a non-transitory computer-readable storage medium (e.g. memory) of the server 106 and executed by a circuit (e.g. using a processor). The circuit can be configured to perform different functions corresponding to different steps of the method described below in connection with FIG. 2. The processor of the circuit can be configured to execute program code portions stored in the memory, the stored program code portions corresponding to different functions. Furthermore, the functions and operations of the circuit can be standalone software applications or form part of a software application that performs additional tasks related to the circuit. Although the described functions and operations can be implemented in software, such functionality can also be performed by dedicated hardware or firmware, or any combination of one or more of hardware, firmware, and software.

[0080] A domain 104 may be associated with an address that specifies how / where to access the domain 104. For example, the address of the domain may be used by an electronic device 110 that is to be located in the coordinate system to access the domain 104, or more specifically, to connect to one of one or more virtual spaces hosted by the domain. The electronic device may obtain the address from a reference object. Alternatively, the electronic device may obtain the address from an application that the electronic device uses to connect to the virtual space. The database 108 may include entries for multiple reference objects 102. Each entry may include a reference object identification, a position and orientation of the corresponding reference object in the coordinate system, and the known physical dimensions and appearance of the corresponding reference object. The reference object identification may be used to access the entry for each reference object. Thus, the position and orientation of the reference object in the coordinate system, and the known physical dimensions and appearance of the reference object may be accessed by sending the reference object identification to the domain. Upon receiving the reference object identification, the domain may send the above information of the reference object back to the electronic device. The electronic device can then determine the relative distance and orientation between itself and the reference object based on the received information and the captured image of the reference object. Finally, the electronic device can determine its position and orientation in a coordinate system based on the determined relative distance and orientation to the reference object. These steps are further described in connection with FIG. 2.

[0081] The database 108 may further include a list of pairs of coordinate system id and coordinate transformation, where the first element of the pair is a coordinate system that indexes a given reference object, and the second element of the pair locates the reference object within the coordinate system. The database 108 may further include possible metadata about the reference object or a pointer thereto. The database 108 may further include a table with rows indexed by unique coordinate system id, columns for a human readable description of the coordinate system, columns for the coordinate system metadata or pointer thereto, and columns containing a list of reference object ids of the reference objects indexed by the coordinate system. The database 108 may further include a table with rows indexed by ordered coordinate system id pairs, where corresponding coordinate systems have a fixed spatial relationship and the corresponding coordinate transformation is known to the database and is encoded in the table columns. The central domain may be online or part of a private network.

[0082] Each entry in the database 108 for a temporary reference object 102 may further include a virtual space identification that identifies the virtual space to which an electronic device already located in the coordinate system is connected. A non-transient reference object 102 may have a predefined virtual space identification associated with it. The virtual space identification may include an address for a domain and a sub-address for a virtual space within the domain.

[0083] Each reference object 102 of the multiple reference objects may further include an address relative to the domain 104, thereby allowing an electronic device 110 that scans the reference object (i.e., captures an image of the reference object) to access the domain 104. The reference object identification may include an address relative to the domain 104, as well as the position and orientation of the reference object 102 within the virtual space.

[0084] A virtual space may have one or more reference objects 102 associated therewith. In other words, one or more reference objects may be used to locate the electronic device 110 in the same virtual space. A reference object 102 may then be associated with a virtual space. In such a case, the reference object 102 may include information about in which of the multiple virtual spaces the electronic device 110 should be located (if the domain hosts multiple virtual spaces). Alternatively, a reference object 102 may be associated with multiple virtual spaces. The multiple virtual spaces associated with a reference object 102 may be different copies of the same virtual space. As an example, multiple copies of the same virtual space may be hosted by the domain 104 to manage multiple users connected to each virtual space via electronic devices, for example by distributing them among different copies of the same virtual space. Alternatively, the multiple virtual spaces associated with a reference object 102 may be completely different virtual spaces. If one reference object 102 is associated with multiple virtual spaces, the electronic device 110 or the domain 104 can determine which virtual space to connect the electronic device to.

[0085] The domain 104 may be distributed across multiple devices. For example, the domain 104 may include a first server that may host a coordinate system and be communicatively connected to a database 108 for managing communications with electronic devices that are to be located within the coordinate system. A second server, which may be part of the same domain or a separate domain, may host a virtual space utilizing the coordinate system hosted by the first server, for example to manage virtual objects in an augmented reality world of the virtual space.

[0086] A domain 104 may contain information about multiple coordinate systems. In such cases, each coordinate system may be identified by a unique id and may further be identified by other associated data, which may include, for example, a human readable description and any other machine readable metadata (e.g., JSON).

[0087] A reference object may fall within the purview of several distinct coordinate systems, in that a device at the reference object would reasonably benefit from knowing its location relative to any one of a given set of local coordinate systems. The domain 104 maintains a list of such coordinate systems for each reference object, and may list several locations. Each of these locations is paired with a coordinate system id in response to a query regarding the location of the reference object. As described in the method of the inventive concept, a device receiving this response is then free to calculate its own location in any of these coordinate systems.

[0088] When a reference object is related to a coordinate system in this way, it can be said that the reference object "indexes" the coordinate system, and also that a given coordinate system "indexes" the reference object.

[0089] A domain can provide API calls to access additional data about a coordinate system named by a unique coordinate system id. Such information can include a human-readable description (or name) of the coordinate system and any metadata associated with the coordinate system. The metadata can be divided into cells or volumetric chunks covering the coordinate system, allowing access on an individual cell basis, e.g. to avoid unnecessarily large download volumes. Other indexing schemes can also be envisaged, e.g. allowing to specify different resolutions for data belonging to the same cell, or for cells to include subdivisions based on their own cells.

[0090] The metadata associated with the coordinate system may include terrain information, geographic landmarks and human readable names, local texture, surface normals, or lighting information, the location or estimated location of broadcasting electronic devices such as WiFi routers or Ultra Wideband (UWB) devices on the map (which may help electronic devices to further site themselves), the location of indexed reference objects, GPS coordinates of landmarks or reference objects, etc. The metadata may include several files, each with a respective type extension. The API may provide sub-functionality to view the file names and extensions.

[0091] Also, the domain may maintain a table containing known coordinate transformations between pairs of coordinate systems, for example indexed by pairs of coordinate system ids. Such transformations are known.

[0092] To avoid storing all possible pairs while maintaining a sufficient set of transformations to be able to determine all known transformations, the domain can choose any number of standard efficient methods, for example, maintaining a (spread) tree-like or star-like system of pairs in each cluster of connected coordinate systems, where all coordinate transformations for each pair in the cluster are (theoretically) known.

[0093] A domain can provide API functionality to allow a device to query for known relationships (i.e. coordinate transformations) between two coordinate systems identified, for example, by coordinate system ids, or the absence of one.

[0094] For coordinate systems that have some fixed relationship to a standard local or global positioning format (i.e., are not attached to an object that can move with respect to these latter systems) and for which the fixed relationship is known, the domain can provide convenient API methods that allow a client device to convert coordinates in that system to equivalent coordinates in the standard system, and vice versa, providing a back conversion from coordinates in the standard system to coordinates in the domain system. For example, this would allow a device to convert coordinates in one of the domain's coordinate systems to GPS coordinates and vice versa.

[0095] Once a device has located itself in one or more coordinate systems, it is free to use this information or forward it to third-party APIs as it sees fit. Thus, the positioning protocol is by default decoupled from the potential uses of positioning, making no assumptions a priori on which devices will participate and free to join or leave the domain at any time.

[0096] This allows, inter alia, a device to utilize features of the method to the extent that it wishes, and to reduce the "software footprint" of the method on the device.

[0097] After establishing a position and orientation in the coordinate system, the device can continue to update this known position using built-in sensors, such as built-in SLAM (Simultaneous Localization and Mapping) technology, i.e., accelerometers, gyroscopes, photogrammetry capabilities, odometers, combinations thereof, and pre-programmed software for parsing and combining such, among others.

[0098] To meet further positioning needs, the device may further utilize nearby terrain or other features or other known nearby reference objects downloaded via the domain coordinate system metadata, possibly in combination with other sensors such as SLAM or GPS units, to help track its changing location, and the device may continue to request new (relevant) metadata as its location changes, by using the metadata API functions already described.

[0099] In other words, the device can use the presence of reference objects and online domains to "bootstrap" itself into a situation where it is resiliently and persistently viable to track changing position with high accuracy, even while movement is ongoing, an accuracy that was not achieved without the ongoing movement prior to calibration.

[0100] The system need not consist of only a single domain. In fact, a feature of the system may be the ability to construct entirely private, wall-offed, or independent domains. In addition to security reasons, such a feature may be useful, for example, to physically remove environments with limited online connectivity. Also, a reference object may list multiple domain addresses for a scanning device or application on such a device to choose from.

[0101] Particular coordinate systems or their metadata or portions thereof may be kept private, for example, from devices that do not present an associated authentication key or similar form of valid authentication information. For example, one or more matching public keys that serve to verify the authentication key or keys may be listed in database 108 or embedded within a portion of the metadata.

[0102] For example, a coordinate system indexed with a given reference object may not be listed and may not present an associated authentication key to a device that scans that reference object, or such a coordinate system may be listed but its metadata or certain parts thereof may not be shared.

[0103] Coordinate system metadata can also be stored in remote domains where access permissions may be monitored for added security.

[0104] In particular, these techniques may enable sensitive terrain, for example corresponding to the interior of a home or private business, to be uploaded in a secure manner and made available to the benefit of the system.

[0105] Domains may also facilitate the addition of new reference objects by users of the system.

[0106] 2 is a flow chart illustrating the steps of a method 200 for determining a position and orientation of an electronic device within a coordinate system. Various steps are described in further detail below. Although the steps of method 200 are shown in a particular order, they may be performed in any suitable order, in parallel, and multiple times.

[0107] An image of the reference object is captured by a camera of the electronic device (S202). The image may be a single image of the reference object, or the image may be a series of images, such as a video feed of the reference object.

[0108] From the image of the reference object, an address for accessing the domain is extracted (S204). The reference object may be a machine-readable optical label. Extracting the address or any other information from the reference object may thus be performed by reading the machine-readable optical label by an electronic device. The address to the domain may be encoded in the machine-readable optical label. The domain may include information about the position and orientation of one or more reference objects in a coordinate system, as well as the known physical dimensions and appearance of one or more reference objects.

[0109] A reference object identification may be extracted from the image of the reference object (S206). The reference object identification may be extracted by computing it locally on the electronic device from the captured image.

[0110] For the reference object, the domain is accessed to obtain the position and orientation in the coordinate system, as well as the known physical dimensions and appearance (S208). In other words, the domain can provide the electronic device with the position and orientation in the coordinate system of the reference object.

[0111] Obtaining the location and orientation, and known physical dimensions and appearance (S208) may include transmitting, by the electronic device, the reference object identification to the domain (S220).

[0112] Obtaining the location and orientation, as well as known physical dimensions and appearance (S208) may include scanning the unique id of the reference object and sending it to the domain. The domain's database may include a table with a row indexed by the unique id of the reference object, a column containing the location of the reference object, and another column containing its dimensional data.

[0113] Other data may be associated with the reference object and may be recorded by the database in the same table or in a separate table indexed by the reference object id. For example, the estimated GPS coordinates of the reference object may be stored.

[0114] The domain may include a database having entries for a plurality of reference objects. Each entry may include a reference object identification, a position and orientation in a coordinate system of the corresponding reference object, and known physical dimensions and appearance of the corresponding reference object. Obtaining the position and orientation, and the known physical dimensions and appearance (S208) may further include accessing, by the domain, an entry in the database corresponding to the reference object identification (S222).

[0115] Based on the known physical dimensions and appearance of the reference object and the apparent dimensions and appearance of the reference object captured in the captured image, a relative distance and orientation of the electronic device to the reference object is determined (S210). Determining the relative distance and orientation of the electronic device (S210) can be performed by the electronic device.

[0116] Based on the position and orientation of the reference object in the coordinate system and the determined relative distance and orientation of the electronic device to the reference object, a position and orientation of the electronic device in the coordinate system is determined (S212). Determining the position and orientation of the electronic device in the coordinate system (S212) can be performed by the electronic device.

[0117] The electronic device may be located within the coordinate system at a position and orientation corresponding to the determined position and orientation of the electronic device within the coordinate system (S214). Locating the electronic device within the coordinate system (S214) may be performed by a domain.

[0118] The electronic device may be connected to the virtual space defined by the coordinate system at a determined position and orientation within the virtual space (S218).

[0119] A domain may host multiple virtual spaces. A determination may be made as to which of the multiple virtual spaces to connect the electronic device to (S216). This may be determined by extracting a virtual space identification from a reference object (216), or this may be determined based on information associated with the electronic device, such as user input indicating which application the electronic device uses to locate within the virtual space or which virtual space to connect to (216).

[0120] The steps of this method may be implemented in computer program code portions which perform the method steps when executed on a device having processing capability.

[0121] For example, as discussed above in connection with FIG. 1, some of the steps may be performed by an electronic device and some of the steps may be performed by a server in the domain.

[0122] Moreover, in practicing the claimed invention, those skilled in the art, from a study of the drawings, the disclosure, and the appended claims, will understand and be able to effect modifications to the variations disclosed.

Claims

1. A computer-implemented method (200) for determining a position and orientation of an electronic device within a coordinate system, comprising: Capturing an image of a reference object by a camera of the electronic device (S202); Extracting (S204) addresses for accessing domains from the images of the reference objects, the domains including information on the position and orientation of one or more reference objects in the coordinate system and on known physical dimensions and appearances of the one or more reference objects; Obtaining (S208) the position and orientation in the coordinate system, and the known physical dimensions and appearance of the reference object by accessing the domain; determining (S210) a relative distance and orientation of the electronic device with respect to the reference object based on the known physical dimensions and appearance of the reference object and the apparent dimensions and appearance of the reference object captured in the captured image; determining (S212) the position and orientation of the electronic device in the coordinate system based on the position and orientation of the reference object in the coordinate system and the determined relative distance and orientation of the electronic device to the reference object; the address is a link or pointer to access the domain, A computer-implemented method, wherein the domain is an online database or centralized hub accessed via a proprietary or public application programming interface or API.

2. The method (200) of claim 1, further comprising: connecting (S218) the electronic device to a virtual space defined by the coordinate system at the determined position and orientation within the coordinate system.

3. The domain hosts a plurality of virtual spaces; 3. The method (200) of claim 2, further comprising: determining (S216) which of the plurality of virtual spaces to connect the electronic device to.

4. extracting (S206) a reference object identification from the image of the reference object; The method (200) of any of claims 1 to 3, wherein obtaining (S208) the position and orientation, and the known physical dimensions and appearance, comprises transmitting (S220) the reference object identification to the domain.

5. the domain includes a database having entries for a plurality of reference objects; each entry includes a reference object identification, a position and orientation of said corresponding reference object within said coordinate system, and known physical dimensions and appearance of said corresponding reference object; 5. The method (200) of claim 4, wherein obtaining the location and orientation, and the known physical dimensions and appearance (S208) further comprises accessing the entry in the database corresponding to the reference object identification (S222).

6. A system (100) for enabling positioning of an electronic device (110) within a coordinate system, comprising: A plurality of reference objects (102), each having a reference object identification and known physical dimensions and appearance; a domain (104) hosting said coordinate system, the domain (104) has an address associated therewith and includes a database (108) having entries for the plurality of reference objects; each entry includes the reference object identification, the position and orientation of the corresponding reference object in the coordinate system, and the known physical dimensions and appearance of the corresponding reference object; Each reference object (102) further includes the address for the domain (104); the address is a link or pointer to access the domain, The domain is an online database or centralized hub accessed through a proprietary or public application programming interface or API.

7. The system (100) of claim 6, wherein the plurality of reference objects (102) are reference objects positioned at different locations in the real world.

8. The system (100) of claim 7, wherein the plurality of reference objects (102) are non-transient or persistent.

9. The system (100) of any of claims 6 to 8, wherein the plurality of reference objects (100) are machine-readable optical labels.

10. The system (100) of any of claims 6 to 9, wherein the domain (108) hosts a plurality of virtual spaces all defined by the coordinate system.

11. The system (100) of claim 10, wherein the domains (104) are configured to track multiple electronic devices in each virtual space.

12. 12. The system (100) of claim 6, wherein the domain is configured to track multiple coordinate systems, including coordinate transformations between pairs of the multiple coordinate systems.

13. The system (100) further includes a temporary reference object (102) displayed on a display (116) of an electronic device positioned within the coordinate system; the temporary reference object (102) has a reference object identification, known physical dimensions and appearance, and said address relative to said domain (104); the database (108) further comprises an entry for the temporary reference object (102); A system (100) as claimed in any one of claims 6 to 12, wherein the entries include the reference object identification of the temporary reference object (102) displayed on the display (116) of the electronic device (116), the position and orientation of the temporary reference object (102) within the coordinate system, and the known physical dimensions and appearance of the temporary reference object.

14. A non-transitory computer readable recording medium having program code portions recorded thereon, A non-transitory computer readable storage medium, the program code portions being adapted to perform the method (200) of any one of claims 1 to 5 when executed on a device having processing capabilities.

Images