Determining mapping between coordinate systems

Abstract

Disclosed is a method comprising receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display; and interacting with the at least one robot by utilizing the mapping.

Description

[0001] DESCRIPTION

[0002] TITLE

[0003] DETERMINING MAPPING BETWEEN COORDINATE SYSTEMS

[0004] TECHNICAL FIELD

[0005] The following example embodiments relate to head-mounted displays and to robots.

[0006] BACKGROUND

[0007] A head-mounted display may be used to assist a user with observing and / or controlling a robot. However, the robot and the head-mounted display may have their own localization and coordinate systems. Thus, there is a challenge in how to enable collaboration between the head-mounted display and the robot.

[0008] SUMMARY

[0009] The scope of protection sought for various example embodiments is set out by the claims. The example embodiments and features, if any, described in this specification that do not fall under the scope of the claims are to be interpreted as examples useful for understanding various embodiments.

[0010] According to an aspect, there is provided an apparatus comprising at least one processor, and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to: receive, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtain one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determine a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interact with the at least one robot by utilizing the mapping.

[0011] According to another aspect, there is provided an apparatus comprising at least one processor, and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to: receive, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtain one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, while the at least one hand of the user is placed on two or more points on the at least one robot, wherein the one or more locations of the at least one hand of the user comprise the two or more points on the at least one robot in the coordinate system of the head-mounted display; determine, based on the two or more points, a spatial relationship between the head-mounted display and the at least one robot in the coordinate system of the head-mounted display; determine a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination of the mapping is based on the spatial relationship, the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interact with the at least one robot by utilizing the mapping.

[0012] According to another aspect, there is provided an apparatus comprising means for causing the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

[0013] According to another aspect, there is provided an apparatus comprising means for causing the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, while the at least one hand of the user is placed on two or more points on the at least one robot, wherein the one or more locations of the at least one hand of the user comprise the two or more points on the at least one robot in the coordinate system of the head-mounted display; determining, based on the two or more points, a spatial relationship between the head-mounted display and the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination of the mapping is based on the spatial relationship, the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

[0014] According to another aspect, there is provided a method comprising: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the headmounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping. According to another aspect, there is provided a method comprising: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, while the at least one hand of the user is placed on two or more points on the at least one robot, wherein the one or more locations of the at least one hand of the user comprise the two or more points on the at least one robot in the coordinate system of the head-mounted display; determining, based on the two or more points, a spatial relationship between the head-mounted display and the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination of the mapping is based on the spatial relationship, the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

[0015] According to another aspect, there is provided a computer program comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping. According to another aspect, there is provided a computer program comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, while the at least one hand of the user is placed on two or more points on the at least one robot, wherein the one or more locations of the at least one hand of the user comprise the two or more points on the at least one robot in the coordinate system of the headmounted display; determining, based on the two or more points, a spatial relationship between the head-mounted display and the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination of the mapping is based on the spatial relationship, the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

[0016] According to another aspect, there is provided a computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

[0017] According to another aspect, there is provided a computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, while the at least one hand of the user is placed on two or more points on the at least one robot, wherein the one or more locations of the at least one hand of the user comprise the two or more points on the at least one robot in the coordinate system of the head-mounted display; determining, based on the two or more points, a spatial relationship between the head-mounted display and the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination of the mapping is based on the spatial relationship, the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

[0018] According to another aspect, there is provided a non-transitory computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the headmounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

[0019] According to another aspect, there is provided a non-transitory computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, while the at least one hand of the user is placed on two or more points on the at least one robot, wherein the one or more locations of the at least one hand of the user comprise the two or more points on the at least one robot in the coordinate system of the head-mounted display; determining, based on the two or more points, a spatial relationship between the head-mounted display and the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination of the mapping is based on the spatial relationship, the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

[0020] BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In the following, various example embodiments will be described in greater detail with reference to the accompanying drawings, in which

[0022] FIG. 1A illustrates an example of an apparatus;

[0023] FIG. IB illustrates an example of an apparatus;

[0024] FIG. 2 illustrates a signal flow diagram;

[0025] FIG. 3 illustrates a flow chart;

[0026] FIG. 4 illustrates a flow chart; and

[0027] FIG. 5 illustrates some examples of visualizations that may be superimposed on a head-mounted display.

[0028] DETAILED DESCRIPTION

[0029] The following embodiments are exemplifying. Although the specification may refer to "an", "one", or "some" embodiment(s) in several locations of the text, this does not necessarily mean that each reference is made to the same embodiment^), or that a particular feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments within the scope of the claims. Furthermore, the words "comprising" and "including" should be understood as not limiting the described embodiments to consist of only those features that have been mentioned, and such embodiments may also contain features that have not been specifically mentioned. Reference numbers, in the description and / or in the claims, serve to illustrate the embodiments with reference to the drawings, without limiting the embodiments to these examples only.

[0030] Different devices, such as robots and head-mounted displays, may use their own coordinate systems to track positions and movements. Herein a coordinate system refers to a reference system that uses one or more numbers (coordinates) to uniquely determine the position of a point or a physical object in space.

[0031] Some example embodiments provide an apparatus and a method for registering a head-mounted display 112 and at least one robot 160 in the same spatial coordinate system (global coordinate system) in order to enable interaction between the head-mounted display 112 and the at least one robot 160 in a shared environment.

[0032] In an example embodiment, the at least one robot 160 may send its current location in the coordinate system of the at least one robot 160 to the headmounted display 112. The user 120 may utilize a hand-tracking unit 115 of the head-mounted display 112 to visually define three-dimensional points on the at least one robot 160, wherein these points define the three-dimensional coordinates of the at least one robot 160 in a coordinate system of the head- mounted display 112. Based on this information, a mapping can then be determined between the coordinate system of the at least one robot 160, and the coordinate system of the head-mounted display 112.

[0033] When the head-mounted display 112 and the at least one robot 160 share the same coordinate system, it is possible to visualize data collected by the at least one robot 160 to the user 120 on the head-mounted display 112. The user 120 can also command the at least one robot 160 in shared spatial coordinates, for example to perform measurements, take a photo, or perform any other task. In robot fleet use cases, registering the head-mounted display 112 with one robot may enable the user 120 to collaborate with all the robots in the fleet.

[0034] With hand-tracking, the head-mounted display 112 can be registered with the at least one robot 160 quickly in any environment, and without requiring any preparations beforehand. The registration does not require any markers, objects or images attached to the at least one robot 160. Furthermore, if the surrounding environment is known as a three-dimensional model, such as a building information model (BIM), and it is registered in either the coordinate system of the head-mounted display 112 or the coordinate system of the at least one robot 160, then both actors can interact also with the environment.

[0035] FIG. 1A illustrates a simplified block diagram of an apparatus 100 for registering the head-mounted display 112 and the at least one robot 160 in the same spatial coordinate system. The at least one robot 160 may be configured to operate autonomously (e.g., controlled by artificial intelligence), or the at least one robot 160 may be controlled by the user 120 with a remote controller 150 or other means (e.g., hand gestures of the user 120).

[0036] Herein a robot may refer to, for example (but is not limited to), at least one of: an industrial robot, a collaborative robot (cobot), a humanoid robot, a swarm robot, an autonomous mobile robot (AMR), an automated guided vehicle (AGV), a drone, an unmanned aerial vehicle (UAV), an unmanned surface vehicle (USV), or an unmanned underwater vehicle (UUV).

[0037] A control system for the at least one robot 160 may be defined as including the at least one robot 160, the remote controller 150, and a wireless communications system 152 between the remote controller 150 and the at least one robot 160.

[0038] Alternatively, an autonomous robot may be operated by a computer system and programmed to fly autonomously, without the need for a human operator or the remote controller 150.

[0039] The apparatus 100 comprises an internal data communication interface 108 configured to communicate with the at least one robot 160 (either directly or via the remote controller 150 or another device) according to a mutually agreed communication protocol.

[0040] For example, via the internal data communication interface 108, the apparatus 100 may receive data comprising at least one of: a location (e.g., latitude, longitude and / or altitude) of the at least one robot 160 in a coordinate system of the at least one robot 160, a route travelled by the at least one robot 160, a direction of the at least one robot 160, a speed of the at least one robot 160, a video feed from the at least one robot 160, battery status, and / or sensor readings such as gyroscope data, magnetometer data, light detection and ranging (LIDAR) data, sonar data, infrared data, barometric pressure data, wind speed and direction data, and / or ambient temperature data, etc.

[0041] The internal data communication interface 108 may be implemented, for example, using a wireless radio transceiver configured to communicate with a wireless transceiver of the remote controller 150 and / or the at least one robot 160. The technologies for the internal data communication interface 108 may include, but are not limited to, one or more of the following: a wireless local area network (WLAN) implemented using an IEEE 802.11 standard or a Wi-Fi protocol suite, a short-range radio network such as Bluetooth or Bluetooth low energy (LE), a cellular radio network employing a subscriber identity module (SIM) or an embedded subscriber identity module (eSIM), or another standard or proprietary wireless connectivity means. Note that in some use cases, the internal data communication interface 108 may additionally or alternatively utilize a standard or proprietary wired connection, such as the universal serial bus (USB) standard.

[0042] The apparatus 100 may also comprise a head-mounted display (HMD) 112 configured to present the user 120 of the apparatus 100 with computergenerated sensory input. For example, at least some of the data associated with the at least one robot 160 may be superimposed (or overlaid) on the head-mounted display 112 in addition to the real-world environment and the at least one robot 160.

[0043] For example, the head-mounted display 112 may comprise an augmented reality (AR) display, or a virtual reality (VR) display, or a mixed reality (MR) display. However, also other applicable implementations of the headmounted display 112 may be used, including, but not limited to: eyeglasses, a heads-up display (HUD), contact lenses with an augmented reality imaging, etc.

[0044] The head-mounted display 112 may be attached to the head of the user 120 with a headband or be helmet-mounted and worn as a visor in front of the eyes of the user 120. In one example, the head-mounted display 112 may be implemented as a see-through display on which holographic images may be displayed. In another example, the head-mounted display 112 may employ cameras to intercept the real-world view and display an augmented view of the real world as a projection.

[0045] The head-mounted display 112 may comprise a six degrees of freedom (6DOF) tracking system configured to track the location and orientation of an object (e.g., the head-mounted display 112 and / or the head of the user 120) in a coordinate system of the head-mounted display 112 with six types of movement. These six types of movements may be categorized into three translations (movements) along the X, Y, and Z axes, and three rotations (orientations) around those three axes.

[0046] The head-mounted display 112 may comprise an integrated handtracking unit 115 configured to track the locations and movements of at least one hand (i.e., one or both hands) of the user 120 in a coordinate system of the headmounted display 112. That is, the hand-tracking unit 115 may be configured to accurately capture, interpret, and translate the hand movements and gestures of the user 120 into corresponding inputs or actions within a virtual, augmented, or mixed reality environment. The hand-tracking unit 115 may utilize a combination of sensors, cameras, and / or infrared technology strategically positioned on or within the HMD 112 to provide real-time detection and analysis of the hands and / or fingers of the user 120.

[0047] The hand-tracking unit 115 may utilize algorithms for gesture recognition, spatial positioning, and motion tracking to enable intuitive user interactions with virtual objects, interfaces, or controls without the need for physical controllers. The hand-tracking unit 115 may support a variety of hand gestures, including but not limited to pinching, grabbing, swiping, and pointing.

[0048] The head-mounted display 112 may further comprise at least one of: a depth camera, a video camera, projection lenses, an inertial measurement unit (e.g., including an accelerometer, a gyroscope, and a magnetometer), and a rechargeable battery. Note that these parts are not illustrated in FIG. 1A or FIG. IB.

[0049] The apparatus 100 may further comprise an immersive reality (1R) client (or engine) 144 configured to handle the basic operations related to integrating real-world views with digital content. The 1R client 144 refers to a software application or system that enables the user 120 to experience immersive reality. For example, the 1R client may be configured to superimpose or overlay digital information, such as images, videos, three-dimensional models, or data, onto the real-world environment as seen through the head-mounted display 112. The 1R client 144 may be a stand-alone software application or it can be embedded with other software such as an internet browser, or it can be part of the operating system of the apparatus 100. The immersive reality client 144 may comprise an augmented reality client, or a virtual reality client, or a mixed reality client.

[0050] The apparatus 100 comprises at least one processor 102 (e.g., including a system on a chip, a custom-made holographic processing unit, and a coprocessor), and at least one memory 104 storing instructions 106 (e.g., computer program code or software) which, when executed by the at least one processor 102, cause the apparatus 100 at least to perform a method or an algorithm 130, for example as shown in FIG. 2, FIG. 3, or FIG. 4.

[0051] The term 'processor' 102 refers to a device that is capable of processing data. In one example, the processor 102 may be implemented as a microprocessor implementing functions of a central processing unit (CPU) on an integrated circuit. The CPU is a logic machine executing the computer program code 106. The CPU may comprise a set of registers, an arithmetic logic unit (ALU), and a control unit (CU). The control unit is controlled by a sequence of the computer program code 106 transferred to the CPU from the (working) memory 104. The control unit may contain a number of microinstructions for basic operations. The implementation of the microinstructions may vary, depending on the CPU design. The one or more processors 102 may be implemented as cores of a single processor and / or as separate processors.

[0052] The term 'memory' 104 refers to a device that is capable of storing data run-time (i.e., working memory) or permanently (i.e., non-volatile memory). The working memory and the non-volatile memory may be implemented by a randomaccess memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), a flash memory, a solid-state disk (SSD), PROM (programmable read-only memory), a suitable semiconductor, or any other means of implementing an electrical computer memory.

[0053] The computer program code 106 is implemented by software. In an embodiment, the software may be written in a suitable programming language, and the resulting executable code may be stored in the at least one memory 104 and executed by the at least one processor 102.

[0054] The computer program code 106 implements the method or algorithm 130. The computer program code 106 may be coded as a computer program (or software) using a programming language, which may be a high-level programming language, such as (but not limited to) C, C++, or Rust, for example. The computer program code 106 may be in source code form, object code form, executable file, or in some intermediate form, but for use in the at least one processor 102 it is in an executable form as an application 144. There are many ways to structure the computer program code 106: the operations may be divided into modules, subroutines, methods, classes, objects, applets, macros, etc., depending on the software design methodology and the programming language used. In modern programming environments, there are software libraries, i.e., compilations of ready-made functions, which may be utilized by the computer program code 106 for performing a wide variety of standard operations. In addition, an operating system (such as a general-purpose operating system) may provide the computer program code 106 with system services.

[0055] One example embodiment provides a computer-readable medium 170 storing the computer program code 106, which, when loaded into the at least one processor 102 and executed by the at least one processor 102, causes the at least one processor 102 to perform the method or algorithm 130, for example as shown in FIG. 2, FIG. 3, or FIG. 4. The computer-readable medium 170 may comprise at least the following: any entity or device capable of carrying the computer program code 106 to the at least one processor 102, a record medium, a computer memory, a read-only memory, an electrical carrier signal, a telecommunications signal, and a software distribution medium. In some jurisdictions, depending on the legislation and the patent practice, the computer-readable medium 170 may not be the telecommunications signal. In an embodiment, the computer-readable medium 170 may be a computer-readable storage medium. In an embodiment, the computer-readable medium 170 may be a non-transitory computer-readable storage medium.

[0056] As shown in FIG. 1A, the computer-readable medium 170 may carry the computer program code 106 as the executable application 144 for the apparatus 100, and / or as an executable application 142 for the at least one robot 160 to transmit the data associated with the at least one robot 160 to the apparatus 100. In some robot environments, a software development kit may be used for the application 142 to interface with the at least one robot 160.

[0057] FIG. 1A illustrates the apparatus 100 as an integrated unit comprising the head-mounted display 112, the at least one processor 102, and the at least one memory 104 storing the instructions 106.

[0058] However, in another embodiment illustrated in FIG. IB, the headmounted display 112 may not be integrated to the apparatus 100. In FIG. IB, the apparatus 100 may be implemented as a distributed apparatus 100, which is communicatively coupled with the head-mounted display 112, and which comprises the at least one processor 102, and the at least one memory 104 storing the instructions 106.

[0059] In the embodiment of FIG. IB, the apparatus 100 may be or comprise any type of computing device external to the head-mounted display 112. For example, the apparatus 100 may be or comprise an edge computing device, or a user apparatus such as a smartphone, a tablet computer or a portable computer carried by the user 120, and the communication coupling may be wired or wireless. Another implementation is such that the apparatus 100 is a networked computer server, which interoperates with the head-mounted display 112 according to a client-server architecture, a cloud computing architecture, a peer-to-peer system, or another applicable distributed computing architecture.

[0060] FIG. 2 illustrates a signal flow diagram according to an example embodiment.

[0061] Referring to FIG. 2, at 201, the user 120 initializes (i.e., starts) the 1R client 144 of the apparatus 100. When the 1R client 144 is running, the user 120 can initialize the process for registering the coordinates of the at least one robot 160 in the coordinate system of the head-mounted display 112 worn by the user 120.

[0062] For example, the 1R client 144 may comprise an augmented reality client. In this case, the head-mounted display 112 may comprise an augmented reality display, and the coordinate system of the head-mounted display 112 may comprise an augmented reality coordinate system of the augmented reality display.

[0063] As another example, the 1R client 144 may comprise a virtual reality client. In this case, the head-mounted display 112 may comprise a virtual reality display, and the coordinate system of the head-mounted display 112 may comprise a virtual reality coordinate system of the virtual reality display.

[0064] At 202, the 1R client 144 establishes communication with the at least one robot 160 and transmits, to the at least one robot 160, a request for a location of the at least one robot 160 in a coordinate system of the at least one robot 160.

[0065] The coordinate system of the at least one robot 160 refers to a spatial framework or reference system used by the at least one robot 160 to understand its position and movements within its environment. The coordinate system of the head-mounted display 112 refers to a spatial framework or reference system used by the head-mounted display 112 to track the position and movements of the head and / or hands of the user 120, and by extension, the user's viewpoint within the immersive environment.

[0066] At 203, in response to transmitting the request of 202, the 1R client 144 receives, from the at least one robot 160, information indicating the location (e.g., three-dimensional coordinates) of the at least one robot 160 in the coordinate system of the at least one robot 160.

[0067] At 204, the 1R client 144 initializes the hand-tracking unit 115 of the head-mounted display 112.

[0068] At 205, when initialized, the hand-tracking unit 115 starts tracking at least one hand (i.e., one or both hands) of the user 120 in the coordinate system of the head-mounted display 112.

[0069] At 206, the 1R client 144 receives a user input from the user 120, the user input indicating to obtain one or more locations of at least one hand (i.e., one or both hands) of the user 120 in the coordinate system of the head-mounted display 112. The user input may comprise, for example, a hand gesture (e.g., a pinch, swipe, grab, or point gesture) detected by the hand-tracking unit 115, or pressing a button, or a voice command, or any other type of user input.

[0070] At 207, based on receiving the user input of 206, the 1R client 144 transmits, to the hand-tracking unit 115, a request for obtaining the one or more locations of the at least one hand of the user 120 in the coordinate system of the head-mounted display 112.

[0071] At 208, in response to receiving the request of 207, the hand-tracking unit 115 obtains or determines, based on the tracking, one or more locations (e.g., three-dimensional coordinates) of the at least one hand of the user 120 in the coordinate system of the head-mounted display 112. The one or more locations of the at least one hand of the user 120 are indicative of a location of the at least one robot 160 in the coordinate system of the head-mounted display 112. In other words, the at least one hand of the user 120 may be placed on the chassis or body of the at least one robot 160, and thus the location of the at least one hand of the user 120 corresponds to the location of the at least one robot 160.

[0072] For example, the user 120 may place their left hand on one point on the at least one robot 160, and their right hand on another point on the at least one robot 160. In this case, the one or more locations may comprise at least a location of the left hand of the user 120, and a location of the right hand of the user 120, wherein the location of the left hand corresponds to one point on the at least one robot 160, and the location of the right hand corresponds to another point on the at least one robot 160.

[0073] Alternatively, the user 120 may consecutively place one hand (or finger) on at least two different points on the at least one robot 160 at different time instants, in which case the one or more locations may comprise multiple locations of the one hand (or finger) of the user 120 at different time instants.

[0074] At 209, the hand-tracking unit 115 transmits, to the 1R client 144, information indicating the one or more locations of the at least one hand of the user 120 in the coordinate system of the head-mounted display 112. The 1R client 144 receives the information.

[0075] At 210, the 1R client 144 determines a mapping between the coordinate system of the at least one robot 160 and the coordinate system of the headmounted display 112, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display 112, and the location of the at least one robot 160 in the coordinate system of the at least one robot 160.

[0076] Herein the mapping refers to aligning or correlating the coordinate system used by the at least one robot 160 with the coordinate system used by the head-mounted display 112. This may involve creating a relationship or translation guide between the coordinate system of the at least one robot 160 and the coordinate system of the head-mounted display 112. This mapping allows the immersive reality system to understand how movements and positions within the coordinate system of the at least one robot 160 correspond to those within the coordinate system of the head-mounted display 112, and vice versa. For example, the one or more locations of the at least one hand may indicate two or more points (e.g., three-dimensional points) on the at least one robot 160 in the coordinate system of the head-mounted display 112. The 1R client 144 may determine, based on the two or more points, a spatial relationship (pose) between the head-mounted display 112 and the at least one robot 160 in the coordinate system of the head-mounted display 112. The mapping may then be determined based at least on the spatial relationship.

[0077] For example, the spatial relationship may refer to a location and an orientation of the head-mounted display 112 relative to the at least one robot 160, or a location and an orientation of the at least one robot 160 relative to the headmounted display 112. The spatial relationship may involve six degrees of freedom (6DOF): three for position (X, Y, Z coordinates) and three for orientation (yaw, pitch, roll angles). By understanding the spatial relationship between the headmounted display 112 and the at least one robot 112, it is possible to accurately map the interactions within the virtual or augmented reality environment to real-world movements and actions of the at least one robot 160.

[0078] As an example, the user 120 may place both of their hands on two different points (i.e., each hand at one point) on the chassis or body of the robot 160, and the locations of the hands at these two points may be registered in the coordinate system of the head-mounted display 112. Alternatively, the user 120 may place a single hand consecutively on two different points on the chassis or body of the robot 160 (i.e., the user 120 may move the hand from one point to the other), and the location of the hand may be registered at each of the two points in the coordinate system of the head-mounted display 112. These two points may define a single vector. For example, if these points are on the robot's body in such a way that they could represent a "forward" direction of the robot, this vector can be considered as the robot's forward vector. The other two vectors may be assumed to be perpendicular to the forward vector. In a three-dimensional space, if one vector (e.g., forward vector) is defined, the other two vectors can be assumed to be to the sides and vertical (up / down), forming a perpendicular set of axes. With the forward vector defined by the two points, it may be assumed that there is no roll (rotation around the forward axis) or tilt (pitch, rotation around the side axis), simplifying the initial orientation estimation.

[0079] By defining more than two points on the robot, a more accurate estimation of the spatial relationship can be achieved. More points allow for better definition of the robot's orientation in space and can help in determining the position and orientation of the head-mounted display relative to the robot more precisely. Additional points can also be used to define the robot's origin or reference point more accurately, which serves as a basis for determining the spatial relationship (pose).

[0080] The 1R client 144 may then interact with the at least one robot 160 by utilizing the mapping, as described below at 211 to 216.

[0081] At 211, the 1R client 144 receives a user input from the user 120, the user input comprising a command for the at least one robot 160 in the coordinate system of the head-mounted display 112. The user input may comprise, for example, a hand gesture detected by the hand-tracking unit 115, or pressing a button, or a voice command, or any other type of user input.

[0082] For example, the command may indicate the at least one robot 160 to perform at least one of: to perform one or more measurements, to take one or more photos, to record a video, or to move to a location specified in the command.

[0083] At 212, the 1R client 144 transforms, based on the determined mapping, the command from the coordinate system of the head-mounted display 112 to the coordinate system of the at least one robot 160.

[0084] At 213, the 1R client 144 transmits the command to the at least one robot 160 in the coordinate system of the at least one robot 160. The at least one robot 160 may then perform the action(s) indicated in the command.

[0085] At 214, the 1R client 144 receives data from the at least one robot 160 in the coordinate system of the at least one robot 160.

[0086] At 215, the 1R client 144 transforms, based on the determined mapping, the data from the coordinate system of the at least one robot 160 to the coordinate system of the head-mounted display 112. At 216, the IR client 144 superimposes, on the head-mounted display 112, one or more visualizations associated with the at least one robot 160 based on the data transformed to the coordinate system of the head-mounted display 112.

[0087] For example, the data may indicate a new location of the at least one robot 160 in the coordinate system of the at least one robot 160, and the one or more visualizations may indicate at least the new location of the at least one robot 160 in the coordinate system of the head-mounted display 112. An example of such a visualization 501 is shown in FIG. 5. The new location refers to a location different from the location received at 203. For example, the new location may refer to a current location of the least one robot 160.

[0088] As another example, the data may indicate a route travelled by the at least one robot 160 in the coordinate system of the at least one robot 160, and the one or more visualizations may indicate at least the route travelled by the at least one robot 160 in the coordinate system of the head-mounted display 112. An example of such a visualization 502 is shown in FIG. 5. For example, the data indicating the route may comprise a set of three-dimensional coordinates indicating multiple past locations of the at least one robot 160.

[0089] FIG. 3 illustrates a flow chart according to an example embodiment of a method performed by the apparatus 100.

[0090] Referring to FIG. 3, in block 301, the apparatus 100 receives, from at least one robot 160, information indicating a location of the at least one robot 160 in a coordinate system of the at least one robot 160.

[0091] The apparatus 100 may transmit, to the at least one robot 160, a request for the location of the at least one robot 160 in the coordinate system of the at least one robot, wherein the information indicating the location of the at least one robot 160 in the coordinate system of the at least one robot may be received in response to transmitting the request.

[0092] In block 302, the apparatus 100 obtains one or more locations of at least one hand of a user 120 in a coordinate system of a head-mounted display 112 worn by the user 120, wherein the one or more locations of the at least one hand of the user 120 are indicative of a location of the at least one robot 160 in the coordinate system of the head-mounted display 112.

[0093] The apparatus 100 may comprise or be connected to a hand-tracking unit 115 configured to track the at least one hand of the user 120 in the coordinate system of the head-mounted display 112. The one or more locations of the at least one hand of the user 120 may be obtained based on the tracking. In other words, the one or more locations of the at least one hand of the user 120 may be received from the hand-tracking unit 115.

[0094] The apparatus 100 may receive a user input from the user 120, the user input indicating to obtain the one or more locations of the at least one hand of the user 120 in the coordinate system of the head-mounted display. The one or more locations of the at least one hand of the user 120 may be obtained in response to receiving the user input.

[0095] The apparatus 100 may comprise the head-mounted display 112, or the apparatus 100 may comprise a computing device external to the head-mounted display (but communicatively coupled to the head-mounted display 112).

[0096] The head-mounted display 112 may comprise an augmented reality display, and the coordinate system of the head-mounted display 112 may comprise an augmented reality coordinate system.

[0097] Alternatively, the head-mounted display 112 may comprise a virtual reality display, and the coordinate system of the head-mounted display 112 may comprise a virtual reality coordinate system.

[0098] In block 303, the apparatus 100 determines a mapping between the coordinate system of the at least one robot 160 and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display 112, and the location of the at least one robot 160 in the coordinate system of the at least one robot 160.

[0099] The one or more locations of the at least one hand may comprise or correspond to two or more points on the at least one robot 160 in the coordinate system of the head-mounted display 112. The apparatus 100 may determine, based on the two or more points, a spatial relationship between the head-mounted display 112 and the at least one robot 160 in the coordinate system of the headmounted display 112. The mapping may be determined based at least on the spatial relationship.

[0100] In block 304, the apparatus 100 interacts with the at least one robot 160 by utilizing the mapping.

[0101] The apparatus 100 may receive a user input from the user 120, the user input indicating to obtain the one or more locations of the at least one hand of the user 120 in the coordinate system of the head-mounted display 112, wherein the one or more locations of the at least one hand of the user 120 may be obtained in response to receiving the user input.

[0102] The one or more locations of the at least one hand may comprise or correspond to two or more points on the at least one robot 160 in the coordinate system of the head-mounted display 112, the two or more points indicating the location of the at least one robot 160 in the coordinate system of the head-mounted display 112. The apparatus 100 may determine, based on the two or more points, a spatial relationship between the head-mounted display 112 and the at least one robot 160 in the coordinate system of the head-mounted display 112, wherein the mapping may be determined based at least on the spatial relationship.

[0103] For example, the interaction may comprise at least: receiving a user input from the user 120, the user input comprising a command for the at least one robot 160 in the coordinate system of the head-mounted display 112; transforming, based on the determined mapping, the command from the coordinate system of the head-mounted display 112 to the coordinate system of the at least one robot 160; and transmitting the command to the at least one robot 160 in the coordinate system of the at least one robot 160.

[0104] The command may indicate the at least one robot 160 to perform at least one of: to perform one or more measurements, to take one or more photos, to record a video, or to move to a location specified in the command.

[0105] Alternatively, or additionally, the interaction may comprise at least: receiving data from the at least one robot 160 in the coordinate system of the at least one robot 160; transforming, based on the determined mapping, the data from the coordinate system of the at least one robot 160 to the coordinate system of the head-mounted display 112; and superimposing, on the head-mounted display 112, one or more visualizations associated with the at least one robot 160 based on the data transformed to the coordinate system of the head-mounted display 112.

[0106] For example, the data may indicate a new location of the at least one robot 160 in the coordinate system of the at least one robot 160, wherein the one or more visualizations may indicate at least the new location of the at least one robot 160 in the coordinate system of the head-mounted display 112.

[0107] As another example, the data may indicate a route travelled by the at least one robot 160 in the coordinate system of the at least one robot 160, wherein the one or more visualizations may indicate at least the route travelled by the at least one robot 160 in the coordinate system of the head-mounted display 112.

[0108] FIG. 4 illustrates a flow chart according to an example embodiment of a method performed by the apparatus 100. The method starts in block 400 and ends in block 416.

[0109] In block 401, the apparatus 100 establishes a connection to at least one robot 160.

[0110] In block 402, the apparatus 100 transmits, to the at least one robot 160, via the connection, a request for a location of the at least one robot 160 in a coordinate system of the at least one robot 160.

[0111] In block 403, the apparatus 100 receives, from the at least one robot 160, in response to transmitting the request, information indicating the location of the at least one robot 160 in the coordinate system of the at least one robot 160.

[0112] In block 404, the apparatus 100 transmits, to a hand-tracking unit 115, a request for obtaining one or more locations of at least one hand of a user 120 in a coordinate system of a head-mounted display 112 worn by the user 120.

[0113] The apparatus 100 may receive a user input from the user 120, the user input indicating to obtain the one or more locations of the at least one hand of the user 120 in the coordinate system of the head-mounted display 112, wherein the request may be transmitted in response to receiving the user input. The apparatus 100 may comprise the head-mounted display 112, or the apparatus 100 may comprise a computing device external to the head-mounted display (but communicatively coupled to the head-mounted display 112).

[0114] The head-mounted display 112 may comprise an augmented reality display, and the coordinate system of the head-mounted display 112 may comprise an augmented reality coordinate system.

[0115] Alternatively, the head-mounted display 112 may comprise a virtual reality display, and the coordinate system of the head-mounted display 112 may comprise a virtual reality coordinate system.

[0116] In block 405, based on transmitting the request, the apparatus 100 receives, from the hand-tracking unit 115, information indicating one or more locations (e.g., three-dimensional coordinates) of the at least one hand of the user 120 in the coordinate system of the head-mounted display 112.

[0117] The apparatus 100 may comprise or be connected to the hand-tracking unit 115, and the hand-tracking unit 115 may be configured to track the at least one hand of the user 120 and determine, based on the tracking, the one or more locations of the at least one hand of the user 120 in the coordinate system of the head-mounted display 112. The one or more locations of the at least one hand are indicative of a location of the at least one robot 160 in a coordinate system of the head-mounted display 112.

[0118] In block 406, if the one or more locations of the at least one hand are received (block 405: yes), the apparatus 100 stores the one or more locations of the at least one hand of the user 120 in the coordinate system of the head-mounted display 112. For example, the apparatus 100 may store the one or more locations in an internal or external memory.

[0119] Alternatively, if the one or more locations of the at least one hand are not received (block 405: no), then the method may return to block 404 and continue from there.

[0120] In block 407, the apparatus 100 determines whether a sufficient number of locations of the at least one hand have been received from the handtracking unit 115 for defining a vector. For example, the apparatus 100 may determine whether the number of stored locations of the at least one hand is above a threshold (e.g., at least two).

[0121] If a sufficient number of locations of the at least one hand have not been received (block 407: no), then the method may return to block 404 and continue from there.

[0122] In block 408, if a sufficient number of locations of the at least one hand have been received (block 407: yes), the apparatus 100 determines a mapping between the coordinate system of the at least one robot 160 and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the headmounted display 112, and the location of the at least one robot in the coordinate system of the at least one robot 160.

[0123] For example, the one or more locations of the at least one hand may comprise or correspond to two or more points on the at least one robot in the coordinate system of the head-mounted display 112. The apparatus 100 may determine, based on the two or more points, a spatial relationship between the head-mounted display 112 and the at least one robot 160 in the coordinate system of the head-mounted display 112. The mapping may be determined based at least on the spatial relationship.

[0124] The apparatus 100 then interacts with the at least one robot 160 by utilizing the mapping, as described below with reference to blocks 409 to 414.

[0125] In block 409, the apparatus 100 receives data from the at least one robot 160 in the coordinate system of the at least one robot 160.

[0126] In block 410, the apparatus 100 transforms, based on the determined mapping, the data from the coordinate system of the at least one robot 160 to the coordinate system of the head-mounted display 112.

[0127] In block 411, the apparatus 100 superimposes, on the head-mounted display 112, one or more visualizations associated with the at least one robot 160 based on the data transformed to the coordinate system of the head-mounted display 112. For example, the data may indicate a new location of the at least one robot 160 in the coordinate system of the at least one robot 160, wherein the one or more visualizations may indicate at least the new location of the at least one robot 160 in the coordinate system of the head-mounted display 112.

[0128] As another example, the data may indicate a route travelled by the at least one robot 160 in the coordinate system of the at least one robot 160, wherein the one or more visualizations may indicate at least the route travelled by the at least one robot 160 in the coordinate system of the head-mounted display 112.

[0129] In block 412, the apparatus 100 receives a user input from the user 120, the user input comprising a command for the at least one robot 160 in the coordinate system of the head-mounted display 112.

[0130] In block 413, the apparatus 100 transforms, based on the determined mapping, the command from the coordinate system of the head-mounted display 112 to the coordinate system of the at least one robot 160.

[0131] In block 414, the apparatus 100 transmits the command to the at least one robot 160 in the coordinate system of the at least one robot 160.

[0132] The command may indicate the at least one robot 160 to perform at least one of: to perform one or more measurements, to take one or more photos, to record a video, or to move to a location specified in the command.

[0133] In block 415, the apparatus 100 determines whether a request for ending the process has been received (e.g., from the user 120).

[0134] In block 416, if the request for ending the process has been received (block 415: yes), then the method ends.

[0135] Alternatively, if the request for ending the process has not been received, then the method may return to block 409 and continue from there.

[0136] The blocks and related functions described above by means of FIG. 2, FIG. 3 and FIG. 4 are in no absolute chronological order, and some of them may be performed simultaneously or in an order differing from the described one. Other functions can also be executed between them or within them, and other information may be sent, and / or other rules applied. Some of the blocks or part of the blocks or one or more pieces of information can also be left out or replaced by a corresponding block or part of the block or one or more pieces of information.

[0137] As used herein, "at least one of the following: " and "at least one of " and similar wording, where the list of two or more elements are joined by "and" or "or", mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0138] FIG. 5 illustrates an example, wherein the apparatus 100 is caused to superimpose, on the head-mounted display 112, one or more visualizations 501, 502 associated with the at least one robot 160 based on the data transformed to the coordinate system of the head-mounted display 112 (e.g., at 216 of FIG. 2, or in block 411 of FIG. 4).

[0139] For example, the data may indicate a new location of the at least one robot 160 in the coordinate system of the at least one robot 160, and the apparatus 100 may be caused to superimpose, on the head-mounted display 112, a visualization 501 indicating the new location of the at least one robot 160 in the coordinate system of the head-mounted display 112.

[0140] As another example, the data may indicate a route travelled by the at least one robot 160 in the coordinate system of the at least one robot 160, and the apparatus 100 may be caused to superimpose, on the head-mounted display 112, a visualization 502 indicating the route travelled by the at least one robot 160 in the coordinate system of the head-mounted display 112.

[0141] The one or more visualizations 501, 502 may be anchored to the coordinate system of the head-mounted display 112, such that the one or more visualizations 501, 502 remain fixed to their coordinates, if the user 120 moves to a different position or turns their head. For example, if the user 120 moves to a new position in the real-world environment, then the one or more visualizations 501, 502 may still be shown in the same location as earlier, but from a different perspective (i.e., from the new position of the user 120).

[0142] As an example, the visualization 501 may comprise a two-dimensional or three-dimensional arrow that may be overlaid to point directly at the new location of the at least one robot 160. As another example, the visualization 501 may comprise a numeric or alphabetic code, such that a number or letter may be superimposed near or on the new location of the at least one robot 160.

[0143] As another example, the visualization 501 may comprise a three- dimensional holographic marker (e.g., symbol or shape).

[0144] As another example, the visualization 501 may comprise a spotlight effect, such that a virtual spotlight or beam of light may be projected onto the new location of the at least one robot 160.

[0145] As another example, the visualization 501 may comprise a floating badge or pin, such that a virtual badge, similar to map pins, may float above the new location of the at least one robot 160.

[0146] As an example, the visualization 502 may comprise a pathway indicator, such that a virtual path, for example dotted lines or footsteps, may be overlaid on the ground to guide the user 120 to the new location of the at least one robot 160.

[0147] It will be obvious to a person skilled in the art that, as technology advances, the inventive concept may be implemented in various ways within the scope of the claims. The embodiments are not limited to the example embodiments described above, but may vary within the scope of the claims. Therefore, all words and expressions should be interpreted broadly, and they are intended to illustrate, not to restrict, the embodiments.

Claims

CLAIMS1. An apparatus comprising at least one processor, and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to: receive, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtain one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determine a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interact with the at least one robot by utilizing the mapping.

2. The apparatus according to claim 1, further being caused to: transmit, to the at least one robot, a request for the location of the at least one robot in the coordinate system of the at least one robot, wherein the information indicating the location of the at least one robot in the coordinate system of the at least one robot is received in response to transmitting the request.

3. The apparatus according to any preceding claim, further being caused to: track the at least one hand of the user in the coordinate system of the head-mounted display,wherein the one or more locations of the at least one hand of the user are obtained based on the tracking.

4. The apparatus according to any preceding claim, further being caused to: receive a user input from the user, the user input indicating to obtain the one or more locations of the at least one hand of the user in the coordinate system of the head-mounted display, wherein the one or more locations of the at least one hand of the user are obtained in response to receiving the user input.

5. The apparatus according to any preceding claim, wherein the one or more locations of the at least one hand comprise two or more points on the at least one robot in the coordinate system of the head-mounted display; wherein the apparatus is further caused to: determine, based on the two or more points, a spatial relationship between the head-mounted display and the at least one robot in the coordinate system of the head-mounted display, wherein the mapping is determined based at least on the spatial relationship.

6. The apparatus according to any preceding claim, wherein the interaction comprises at least: receiving a user input from the user, the user input comprising a command for the at least one robot in the coordinate system of the head-mounted display; transforming, based on the determined mapping, the command from the coordinate system of the head-mounted display to the coordinate system of the at least one robot; and transmitting the command to the at least one robot in the coordinate system of the at least one robot.

7. The apparatus according to any preceding claim, wherein the command indicates the at least one robot to perform at least one of: to perform one or more measurements, to take one or more photos, to record a video, or to move to a location specified in the command.

8. The apparatus according to any preceding claim, wherein the interaction comprises at least: receiving data from the at least one robot in the coordinate system of the at least one robot; transforming, based on the determined mapping, the data from the coordinate system of the at least one robot to the coordinate system of the headmounted display; and superimposing, on the head-mounted display, one or more visualizations associated with the at least one robot based on the data transformed to the coordinate system of the head-mounted display.

9. The apparatus according to claim 8, wherein the data indicates a new location of the at least one robot in the coordinate system of the at least one robot, wherein the one or more visualizations indicate at least the new location of the at least one robot in the coordinate system of the head-mounted display.

10. The apparatus according to any of claims 8 to 9, wherein the data indicates a route travelled by the at least one robot in the coordinate system of the at least one robot, and wherein the one or more visualizations indicate at least the route travelled by the at least one robot in the coordinate system of the head-mounted display.

11. The apparatus according to any preceding claim, wherein the headmounted display comprises an augmented reality display, and the coordinate system of the head-mounted display comprises an augmented reality coordinate system.

12. The apparatus according to any preceding claim, wherein the apparatus comprises the head-mounted display.

13. The apparatus according to any of claims 1 to 11, wherein the apparatus comprises a computing device external to the head-mounted display.

14. A method comprising: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot; obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

15. A non-transitory computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: receiving, from at least one robot, information indicating a location of the at least one robot in a coordinate system of the at least one robot;obtaining one or more locations of at least one hand of a user in a coordinate system of a head-mounted display worn by the user, wherein the one or more locations of the at least one hand of the user are indicative of a location of the at least one robot in the coordinate system of the head-mounted display; determining a mapping between the coordinate system of the at least one robot and the coordinate system of the head-mounted display, wherein the determination is based on the one or more locations of the at least one hand in the coordinate system of the head-mounted display, and the location of the at least one robot in the coordinate system of the at least one robot; and interacting with the at least one robot by utilizing the mapping.

Images