Method and device for positioning IoT device

Abstract

The method for locating an IoT device using AR glasses comprises the following steps: obtaining an indoor map; determining the position and viewing direction of the AR glasses on the indoor map based on initial information of the AR glasses; receiving tracking information related to the IoT device to be located from the AR glasses; and determining the position of the IoT device to be located according to the determined position and viewing direction and in combination with the tracking information of the AR glasses.

Description

Background Art

[0001] The purpose of the present disclosure is to provide a method for locating an Internet of Things (IoT) device using augmented reality (AR) glasses. In addition, the purpose of the present disclosure is to provide a data processing device and software for locating IoT devices.

[0002] Today, more and more objects in the home are connected together and to the Internet. Internet connectivity for IoT devices in the home is usually accomplished through a home network gateway, which is responsible for routing messages across multiple different types of networks (for example, Bluetooth, ZigBee, or Wi-Fi), between devices, or from devices to outside the home (via the Internet).

[0003] From the user's perspective, IoT devices are usually controlled by at least one home automation system that can be accessed both at home (usually via Wi-Fi) and from outside the home (mostly via the Internet). It is not uncommon for multiple IoT devices to belong to different home automation ecosystems, and users sometimes have to switch between different home automation applications or use bridge applications, which are often complex to set up and do not provide all the features of the original home automation application.

[0004] Additionally, as the number of IoT devices in the home continues to increase, quickly operating individual devices from home automation applications becomes increasingly cumbersome.

[0005] The first solution that has been introduced in the prior art is to use a voice control system to control IoT devices.

[0006] In any case, the location of an IoT device is necessary in order to address it correctly. An indication of the location of an IoT device is present to some extent in every home automation system today, but it is always limited to the room classification. For example, a connected light bulb can be identified as part of the living room, but there is no information about where exactly in the living room the light is located. And if there are several lights, it is impossible to distinguish them unless the user has intentionally assigned them different names, which need to be remembered and distributed to all users of the home automation system.

[0007] Furthermore, home automation systems usually have little information about the spatial configuration of the home, at most they know how many rooms there are. Today, only one IoT device has such additional knowledge: a connected vacuum cleaner robot, which usually scans the house to better define its cleaning paths. However, although such a house map or indoor map is generated, it is not shared with other devices or home automation systems. In the future, it is also foreseeable that other IoT devices, such as cameras with Time-of-Flight (ToF) sensors, will also be able to provide or at least help build such indoor maps.

[0008] Therefore, in order to better control IoT devices, the accurate location of IoT devices is required. Positioning, that is, initially determining the location of newly implemented IoT devices and then locating IoT devices in existing locations, becomes the key to efficiently control home automation systems.

[0009] Therefore, an object of the present disclosure is to provide a method and a data processing device for locating an IoT device. Summary of the invention

[0010] In one aspect of the present disclosure, a method for locating an IoT device using augmented reality (AR) glasses is provided, the method comprising the following steps:

[0011] Get indoor maps;

[0012] Determine, based on the initial information of the AR glasses, an AR position of the AR glasses on the indoor map and a viewing direction of the AR glasses;

[0013] Receiving tracking information related to the IoT device to be located from the AR glasses; and

[0014] Based on the determined position and viewing direction and in combination with the tracking information of the AR glasses, the IoT position of the IoT device to be located is determined.

[0015] Among them, augmented reality is a real environment experience in which virtual elements (such as graphics, videos, animations, etc.) are used to represent additional information. Among them, the virtual elements are provided to users through visual perception by AR glasses.

[0016] In addition, IoT devices include various small / personal devices that are connected to the Internet. This may include household devices such as light bulbs, televisions, vacuum cleaners and their bases, sound devices, smart TVs, kitchen appliances, etc., but the present disclosure is not limited thereto. IoT devices can provide different functions that can be controlled through an Internet connection. Typically, a software application is run on the end user to provide a connection to the functions of the IoT device.

[0017] In addition, the indoor map is a map of a home, indicating the relative positions between objects on the indoor map. Therefore, an indoor map is obtained, and the indoor map is used to determine the AR position and viewing direction of the AR glasses on the indoor map, wherein the AR position and viewing direction of the AR glasses are based on the initial information of the AR glasses. Preferably, the AR glasses include at least one sensor to obtain the initial information. The at least one sensor may be a compass, a gyroscope and / or an accelerometer to determine the movement / rotation of the AR glasses, in particular the angle of rotation at least in the horizontal plane. Therefore, through the sensor in the AR glasses, it is preferably possible to determine the angle between the first viewing direction and the second viewing direction as initial information in order to determine the AR position and viewing direction of the AR glasses on the indoor map.

[0018] In order to determine the IoT position of the IoT device, the AR glasses receive tracking information about each IoT device to be located. In order to obtain the tracking information, preferably, the user points the AR glasses at the IoT device by looking at or being in the direction of the IoT device to be located. The IoT position of the IoT device to be located is determined based on the AR position and viewing direction of the AR glasses and in combination with the tracking information about the IoT device to be located obtained by the AR glasses.

[0019] Thus, the AR glasses are used to determine the IoT position of the IoT device on the indoor map. However, in order to be able to determine the IoT position of the IoT device, it is first necessary to determine the AR position and viewing direction of the AR glasses on the indoor map through the initial information of the AR glasses. Therefore, the positioning of the IoT device is simplified, and it is no longer necessary to manually locate the IoT device on the indoor map or to manually assign the position of the IoT device to the IoT device in some way. Therefore, the correctly located IoT device can be more easily addressed for control. In particular, it will be possible to directly indicate the IoT device on the indoor map on the user terminal. Therefore, for example, the same IoT device (two light bulbs in one room) can be easily distinguished and appropriately addressed and controlled by the user.

[0020] Preferably, determining the AR position of the AR glasses on the indoor map and the viewing direction of the AR glasses based on the initial information of the AR glasses includes:

[0021] A user input indicating the AR position and the viewing direction of the AR glasses on the indoor map is received on the terminal.

[0022] Therefore, through user input, the AR position of the AR glasses on the indoor map is provided. The user input may be a touch input or any other click or mark input position on the terminal. Preferably, the AR position is directly indicated on the indoor map displayed by the terminal. In addition, the input of the current viewing direction can be by: indicating an object in the user's current field of view on the terminal; or drawing a direction from the AR position of the AR glasses to the viewing direction on the terminal. Preferably, the viewing direction is directly indicated on the indoor map displayed by the terminal. Therefore, through user input on the terminal, the AR position and viewing direction of the AR glasses can be easily provided, and the AR position and viewing direction of the AR glasses can be used to determine the IoT position of the IoT device to be located.

[0023] Preferably, determining the AR position of the AR glasses on the indoor map and the viewing direction of the AR glasses based on the initial information of the AR glasses includes:

[0024] Receiving tracking information related to at least three anchor points from the AR glasses, wherein the anchor point positions of the at least three anchor points on the indoor map are known;

[0025] For any two anchor points of the at least three anchor points, determine the angle between the two anchor points according to respective tracking information of the two anchor points; and

[0026] The AR position and the viewing direction of the AR glasses are determined according to the multiple angles.

[0027] Preferably, tracking information of at least three anchor points is received from the AR glasses while the AR glasses remain in the same position. Therefore, the position of the AR glasses is fixed, and in order to receive tracking information of at least three anchor points, only the viewing direction of the AR glasses is changed. Among them, the anchor point can be any point in the user's real environment, where the position of the anchor point on the indoor map is known. By viewing one of these anchor points, the AR glasses will receive tracking information. Preferably, as the AR glasses receive the tracking information, multiple angles between at least three anchor points are determined. Therefore, at least two angles are determined, namely, a first angle between the first anchor point and the second anchor point and a second angle between the second anchor point and the third anchor point. Preferably, these angles are determined by an accelerometer or a gyroscope implemented in the AR glasses. From the multiple angles determined, the AR position and viewing direction of the AR glasses can be determined.

[0028] Preferably, determining the AR position of the AR glasses on the indoor map and the viewing direction of the AR glasses based on the initial information of the AR glasses comprises: receiving tracking information related to a first anchor point and at least one second anchor point from the AR glasses, wherein at least the first anchor point is moving, and the anchor point positions of the first anchor point and the at least one second anchor point on the indoor map are known;

[0029] Determine a first angle between the first anchor point and the second anchor point at a first time point;

[0030] determining a second angle between the first anchor point and the second anchor point at a second time point, wherein at least an anchor point position of the first anchor point at the second time point is different from an anchor point position of the first anchor point at the first time point;

[0031] The AR position and the viewing direction of the AR glasses are determined according to the first angle and the second angle.

[0032] Therefore, the first anchor point knows the position of the anchor point on the indoor map while moving. The tracking information of the first anchor point can be used at the first time point and the second time point to determine the first angle and the second angle, wherein the AR position and viewing direction of the AR glasses can be determined from the multiple angles. Therefore, when at least one anchor point with a known position on the indoor map is moving, only two anchor points are needed.

[0033] Preferably, determining the angle between at least three anchor points comprises: determining the direction / sign of an angle between two anchor points. Therefore, the initial information is obtained regardless of whether the angle between the two anchor points is determined by the rotation of the AR glasses from left to right or from right to left. The additional information can be used to determine the position and viewing direction of the AR glasses. Among them, the direction (or mathematical sign) of the angle is preferably determined based on sensor data such as a gyroscope, accelerometer or compass of the AR glasses.

[0034] Preferably, at least one anchor point comprises an AR marker to be recognized by the AR glasses to identify the anchor point. In this way, automatic or at least semi-automatic recognition of the anchor point can be made easier. By means of the AR marker, the anchor point can be uniquely identified. Thus, the position of the anchor point on the indoor map can be assigned to the AR marker automatically recognized by the AR glasses.

[0035] Preferably, the AR glasses identify at least one anchor point by image recognition. In this way, automatic or at least semi-automatic identification of anchor points can be made easier. Among them, anchor points can be provided by unique objects in the home, such as TVs, refrigerators, ovens, etc. These objects can be recognized by the AR glasses and then used as anchor points to determine the position and viewing direction of the AR glasses.

[0036] Preferably, the compass information of the AR glasses is determined, and the indoor map is oriented according to the compass information. Preferably, the AR glasses include a compass to generate compass information, which can provide an orientation for an indoor map that does not usually include any compass information. Then, the indoor map is oriented according to the compass information, and the orientation of the indoor map is fixed for further use. Then, the correctly oriented indoor map can facilitate the subsequent determination of the position and viewing direction of the AR glasses. As a result, the determination of the position and viewing direction of the AR glasses can be simplified, or the accuracy of the position and viewing direction can be improved by adding compass information.

[0037] Preferably, based on the initial information of the AR glasses, determining the AR position of the AR glasses on the indoor map and the viewing direction of the AR glasses includes: using compass information to determine the directions of multiple angles between at least two anchor points and / or the angle between any two of the at least two anchor points.

[0038] Preferably, the anchor point is provided by one or more IoT devices. Therefore, preferably, the IoT devices are marked with AR markers. In particular, these AR markers can be provided by the manufacturer of the IoT device to simplify the positioning of the IoT device. Similarly, the appearance of the IoT device is known to the manufacturer of the IoT device, wherein this information can be used by image recognition in order to automatically identify the IoT device by the AR glasses.

[0039] Preferably, the indoor map is provided by a vacuum cleaner and / or a surveillance camera. Preferably, the surveillance camera is any camera including a time-of-flight camera, a stereo camera or any other optical device providing 3D information, such as light detecting and ranging (LIDAR) or the like.

[0040] Preferably, determining the IoT position of the IoT device to be located based on the determined AR position and viewing direction and in combination with tracking information of the AR glasses comprises: determining a first viewing direction of the AR glasses from the first AR position to the IoT device to be located;

[0041] Determining at least a second viewing direction of the AR glasses from the second AR position to the IoT device to be located; and

[0042] The IoT position of the IoT device to be located on the indoor map is determined according to the first AR position, the first viewing direction, the second AR position, and the second viewing direction.

[0043] Preferably, in particular when compass information is not available, a third viewing direction from the AR glasses to the IoT to be located is also used. In this case, the determination of the IoT position of the IoT device is similar to the steps of determining the AR position of the AR glasses described above and below. Therefore, with AR glasses having a known AR position and viewing direction, it is easier to locate IoT devices of unknown locations on an indoor map. This IoT position of the IoT device can be assigned to the IoT device for controlling the IoT device and / or for storage. Preferably, if the IoT position of the IoT device is determined, the IoT device can be used as an anchor point for further determining the AR position and viewing angle of the AR glasses.

[0044] Preferably, based on the determined AR position and viewing direction and in combination with the tracking information of the AR glasses, the IoT position of the IoT device to be located is determined, including: using compass information to determine the first viewing direction and the second viewing direction.

[0045] In another aspect of the present disclosure, a data processing device is provided, comprising a processor and a memory storing instructions, where the instructions are suitable for executing the method as described above when the instructions are executed by the processor.

[0046] Preferably, the data processing device is connected to the AR glasses to receive tracking information of the AR glasses so as to determine the AR position and viewing direction and / or multiple viewing directions of the AR glasses, thereby locating the IoT device.

[0047] In another aspect, a non-transitory computer-readable storage medium stores instructions for execution by a processor, which, when executed by the processor, causes the processor to perform the method as described above.

[0048] Therefore, through the present disclosure, the positioning of IoT devices is made easier by using AR glasses. Therein, the AR position and viewing direction of the AR glasses on the acquired indoor map are first determined, and the IoT position of the IoT device to be located is determined using this information. Therefore, there is no need to manually indicate the location of the IoT device. Through the known IoT position of the IoT device, the control of the IoT device in the home automation application can be easily controlled, and due to the determined IoT position, the identification of a single IoT device in the home automation application can be simplified. Therefore, even if the same or similar IoT devices appear in a room more than once, these IoT devices can be clearly distinguished by their accurate location. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Reference will now be made, by way of example, to the accompanying drawings which illustrate several exemplary embodiments of the present application, in which:

[0050] Figure 1 is a first embodiment of the method according to the present disclosure;

[0051] Figure 2 is an exemplary indoor map according to the present disclosure;

[0052] Figure 3 is a detailed embodiment of the method according to the present disclosure;

[0053] Figure 4 is a detailed embodiment for determining an AR position and a viewing direction of AR glasses;

[0054] Figure 5 is a detailed embodiment of the method according to the present disclosure;

[0055] Figure 6 is a schematic diagram of determining the AR position and viewing direction of AR glasses according to the present disclosure;

[0056] Figure 7 It is a schematic diagram of the calculation steps of the AR position of the AR glasses;

[0057] Figure 8 is a detailed embodiment of the method according to the present disclosure;

[0058] Fig. 9 is a detailed embodiment of the method according to the present disclosure;

[0059] Fig.10 is an example graph for calculating the IoT position of the IoT device to be located; and

[0060] Fig.11 is a data processing device according to the present disclosure. DETAILED DESCRIPTION

[0061] The present application describes a method for determining the location of an IoT device on an indoor map, and the location of the IoT device is referred to as the IoT location hereinafter. Among them, AR glasses are used to locate the IoT device. In order to be able to determine the IoT location of the IoT device, the first step is to determine the location and viewing direction of the AR glasses on the indoor map. In the following, the location of the AR glasses is referred to as the AR location.

[0062] Some IoT devices already exist in today's home automation systems, which generate indoor maps in order to perform tasks. However, these maps are not shared with other IoT devices and are generally not shared with the home automation system. Therefore, it is not currently known or possible to use the exact location of an IoT device in a home automation system in a way that can be used to control the IoT device.

[0063] One of the IoT devices that is capable of creating indoor maps is the robot vacuum cleaner. In many home automation systems that include a robot vacuum cleaner, the robot has the ability to scan the house (usually using SLAM lasers) and build a map of the house to complete its tasks. Figure 2 The indoor Fig.10 Indoors Fig.10 1, a robot vacuum cleaner 12 and a docking station 14 are shown. In addition, the indoor map reflects the movement trajectory of the robot vacuum cleaner 10 to guide the robot vacuum cleaner to bypass objects while cleaning the entire area. Such a map is usually only used by the robot vacuum cleaner to define the best cleaning path for the house, while allowing the user to clearly indicate the path followed by the vacuum cleaner.

[0064] As the number of IoT devices in the home increases, it becomes interesting to share such maps between all IoT devices and locate each IoT device on the map. This allows users to have a visual access to their devices through home automation applications instead of looking for IoT devices in a room or device list.

[0065] For example, a home contains several light bulbs, a set of speakers, a TV, and a vacuum cleaner, all of which can be operated in a home automation application. When there are multiple light bulbs in a home, operating the light bulbs is usually a cumbersome operation because the user first needs to identify in the application exactly which light bulb he wants to operate.

[0066] Thus, by using a map that indicates the exact location of IoT devices such as light bulbs, the IoT devices can be activated (i.e., turned on) simply by pointing and clicking on the desired IoT devices.

[0067] The present disclosure provides a method for determining the IoT position of an IoT device through AR glasses. However, in order to determine the IoT position of the IoT device on the indoor map, the AR position and viewing direction of the AR glasses on the indoor map must first be determined. Once this is done, since one or more sensors in the AR glasses are capable of tracking the movement of the AR glasses, the AR glasses can be tracked in real time. Such sensors implemented in the AR glasses can be accelerometers, gyroscopes, compasses, or cameras. Preferably, more than one of these sensors, and more preferably, all of these sensors are implemented in combination in the AR glasses.

[0068] exist Figure 1 The method according to the present disclosure is schematically shown in FIG. 1 , and the method comprises the following steps:

[0069] S01: Get indoor map;

[0070] S02: determining an AR position of the AR glasses on the indoor map and a viewing direction of the AR glasses based on initial information of the AR glasses;

[0071] S03: receiving tracking information related to the IoT device to be located from the AR glasses; and

[0072] S04: Determine the IoT position of the IoT device to be located based on the determined AR position and viewing direction and in combination with the tracking information of the AR glasses.

[0073] In the first solution, in order to determine the AR position and viewing direction of the AR glasses on the indoor map, the user can directly input the AR position and viewing direction of the AR glasses on the indoor map on the terminal. The method includes step S021: receiving user input indicating the AR position and viewing direction of the AR glasses on the indoor map on the terminal.

[0074] like Figure 4 The exemplary indoor ground shown Fig.10 , the AR glasses are in an AR position 16 indicated by an “X” facing the vacuum cleaner robot 12. Therefore, the user can display the indoor location on his terminal device through user input (such as click, touch and / or drawing input). Fig.10 The AR position 16 and the viewing direction indicated by the arrow 18 are indicated on the map. The AR position and viewing direction of the AR glasses are thereby known. The changes in the AR position and viewing direction are tracked in real time by the sensors of the AR glasses, and changes in the AR position and / or viewing direction of the AR glasses can be determined. Therefore, once the AR glasses are positioned on the indoor map, combined with the viewing direction, the information can be used to locate the IoT device to determine the IoT position of the IoT device.

[0075] There is no need to determine the absolute position of the AR glasses. The relative position of the AR glasses relative to the indoor map is sufficient.

[0076] Although the use of an indoor map generated by a vacuum cleaner robot is described above, any other indoor map generated by any other device (eg, a surveillance camera, etc.) may be used.

[0077] Reference now Figure 5 A second solution for determining the AR position and viewing direction of AR glasses is schematically shown, comprising the following steps:

[0078] S022: receiving tracking information related to at least three anchor points from the AR glasses, wherein anchor point positions of the at least three anchor points on the indoor map are known;

[0079] S023: Determine the angle between any two anchor points among the at least three anchor points; and

[0080] S024: Determine the AR position and viewing direction according to the multiple angles.

[0081] Therefore, three anchor points are used to determine the AR position and viewing direction of the AR glasses, where the anchor points are located indoors. Fig.10 The position on the ground is known. Among them, the three anchor points can be provided by IoT devices with known positions. In particular, if there is a vacuum cleaner robot 12 in the home automation system, the indoor ground can be Fig.10 The base 14 of the vacuum cleaner robot 12 with a known position on the ground is used as an anchor point; Fig.10 The vacuum cleaner robot 12 with a known position can itself be used as a further anchor point. Therefore, in order to facilitate the determination of the AR position and viewing direction of the AR glasses, only one more anchor point with a known position needs to be added.

[0082] exist Figure 6 The figure schematically depicts a situation with a first anchor point A, a second anchor point B and a third anchor point C, wherein it is known that the indoor ground Fig.10 The positions of the upper anchor points A, B, and C. To determine the AR position of the AR glasses, first determine the angle between anchor point A and anchor point B. , to provide a possible position of the AR glasses on the first arc 20. Angle The determination of may be provided by one or more sensors implemented in the AR glasses. Therefore, the user first points the AR glasses at anchor point A, indicating an object at anchor point A, so as to identify anchor point A and thereby be able to retrieve the position of anchor point A on the indoor map. Then, the user turns the head to point the AR glasses at anchor point B, indicating an object at anchor point B, so as to be able to retrieve the position of anchor point B on the indoor map. By rotating the head and pointing the AR glasses from the first anchor point A to the second anchor point B, the AR glasses are rotated by an angle , the angle is determined and used to determine the AR position and viewing direction of the AR glasses. Preferably, the user remains in the same position.

[0083] Again, determine the angle as described above for anchor points B and C. The same steps are followed to determine the corresponding angles . A possible position of the AR glasses on the second arc 22 is provided, wherein the intersection I of the first arc 20 and the second arc 22 indicates the AR position of the AR glasses. From the AR position of the AR glasses, the viewing direction toward one of the anchor points A, B or C can be determined, so that the AR position and viewing direction of the AR glasses can be fully determined.

[0084] To simplify, Figure 7A schematic situation with only two anchor points is shown, illustrating the mathematical steps for determining the first arc 20. The method for determining the first arc 20 comprises the following steps:

[0085] 1) Figure 7 Point H in is considered to be a point on the median of line segment AB (this means that the connecting line segment HM is perpendicular to line segment AB and intersects line segment AB in the middle of line segment AB. This forms an equilateral triangle △ABH), in which the angle 24 (denoted by ⦠AHB - line segment AH is rotated about point H until it overlaps line segment HB) is equal to The direction of the angle is indicated by arrow 24 , and here and below, the counterclockwise direction is defined as the positive direction.

[0086] 2) Determine the triangle △HBM, where M is located in the middle of AB, and the distance HM is calculated by the following formula:

[0087]

[0088] 3) In triangle △HOP, P is located in the middle of distance HB, and line segment HB is perpendicular to line segment OP. The radius of the first arc 20, that is, distance HO, can be calculated using the following formula:

[0089]

[0090]

[0091] 4) By using Figure 7 Given the relationship: , the distance HM represented by HO is given by:

[0092]

[0093] 5) Then the distance OM can be calculated by the following formula:

[0094]

[0095] Plugging in the formula for HM and HO yields:

[0096]

[0097] So using the trigonometric identities

[0098] and .

[0099] Therefore, knowing the length OMAnd knowing point M, the position of point O can be known. In addition, the radius HO of the first arc 20 passing through points A, B and H is also known. Therefore, the quadratic equation of the first arc 20 can be calculated. Figure 6 As shown, the same steps are repeated for anchor point B and anchor point C to calculate the quadratic equation of the second arc 22. Solving the two quadratic equations for the first arc 20 and the second arc 22 provides two solutions: B and the actual position of the AR glasses represented by the intersection I of the first arc 20 and the second arc 22.

[0100] like Figure 7 As shown, only knowing In the case of the value of , there are two possible arcs 20-1 and 20-2 for the first arc 20. However, if the angle If the direction / sign of Figure 7 As shown, if the angle with direction is determined as , then the arc 20-1 is determined as the first arc 20. If the angle with direction is determined as , then arc 20-2 will uniquely indicate a possible AR position of the AR glasses. Therefore, there is no ambiguity between the first arc 20 and the similar second arc 22 by the direction of the angle. Therefore, the intersection I also provides a unique solution. Preferably, the direction of the angle is determined by a sensor and / or compass information in the AR glasses.

[0101] However, even if the angular direction between the anchor points is unknown, a solution for the AR position of the AR glasses can be determined. If the AR glasses are not located at or at least not close to anchor point B, and the angle ⦠ABC between the anchor points is greater than 0° and less than 180°, then the intersection I of the two arcs 20 and 22 away from anchor point B is the position of the AR glasses.

[0102] Optionally, a fourth anchor point can be used to provide a single solution for the position of the AR glasses on the indoor map.

[0103] Preferably, if one of the anchor points is provided by the robot vacuum cleaner 12 moving in a known position, the robot vacuum cleaner 12 can be used as the first anchor point at a first point in time and as the second anchor point at a second point in time. Figure 8 A method for determining an AR position and a viewing direction of AR glasses is shown, the method comprising the following steps:

[0104] S025: receiving tracking information related to a first anchor point and at least one second anchor point from the AR glasses, wherein at least the first anchor point is moving, and anchor point positions of the first anchor point and the at least one second anchor point on the indoor map are known;

[0105] S026: Determine a first angle between the first anchor point and the second anchor point at a first time point;

[0106] S027: Determine a second angle between the first anchor point and the second anchor point at a second time point, wherein at least an anchor point position of the first anchor point at the second time point is different from at least an anchor point position of the first anchor point at the first time point; and

[0107] S028: Determine the AR position and viewing direction of the AR glasses according to the first angle and the second angle.

[0108] Therefore, in order to facilitate the determination of the AR position and viewing direction of the AR glasses, it is sufficient as long as the vacuum cleaner robot 12 is mobile and the vacuum cleaner robot 12 and the base 14 both have known positions on the indoor map.

[0109] In some embodiments, dedicated AR markers are used on at least one, multiple, or all IoT devices in the home. With such AR markers, the user does not need to identify the IoT device he is currently viewing, as the AR marker will provide that information and uniquely identify the IoT device. Thus, just by browsing the house, the AR glasses will easily detect the IoT devices and recognize them due to the devices' AR markers. Once this is done, the home automation system can determine the position of the AR glasses by mathematical calculations from the IoT positions of the AR-marked IoT devices and the angle of view between them reported by the AR glasses, as described above. In addition, when there is compass information, the IoT position of a single IoT device can also be calculated by viewing the single IoT device from two different positions of the AR glasses; or if only the angle information (between different viewing positions) is known, the IoT position of a single IoT device can also be calculated by viewing the single IoT device from three different positions of the AR glasses.

[0110] The obvious advantage of this solution is that not only is it fully automatic, but it is also completely transparent to the user, provided that there are enough (at least 3) IoT devices in the home that the AR glasses can identify and whose locations on the map are known. The only thing this solution requires is to attach AR markers to the IoT devices. Such AR markers can be provided by the IoT manufacturer or downloaded later.

[0111] In another embodiment of the present disclosure, the above embodiment can also help track the AR glasses in real time at home by always running the positioning of the AR glasses in the background.

[0112] In another embodiment, object recognition technology is used to identify at least one, multiple, or all IoT devices. In fact, for some IoT devices, machine learning technology can be used and the AR glasses software can be trained to recognize specific IoT devices. This has the obvious advantage that it is a fully automatic and completely transparent solution.

[0113] In some embodiments, the AR glasses include a compass to obtain compass information. In a first step, the compass information is connected to the indoor map and the absolute direction of the indoor map relative to the compass information is provided. To this end, when the AR position and viewing direction of the AR glasses are first determined, the compass information about the position is recorded.

[0114] Afterwards, whenever the AR position and viewing direction of the AR glasses are determined, the compass information can be used and synchronized with the orientation map of the house to better position the AR glasses and improve accuracy. In addition, if compass information is present, only two anchor points are required to uniquely identify the AR position of the AR glasses. However, using a third anchor point can bring higher accuracy.

[0115] Fig. 9 An embodiment for locating an IoT device is shown, comprising the following steps:

[0116] S041: Determine a first viewing direction of the AR glasses from the first AR position to the IoT device to be located;

[0117] S042: Determine a second viewing direction of the AR glasses from the second AR position to the IoT device to be located; and

[0118] S043: Determine an IoT position of the IoT device to be located on the indoor map according to the first AR position, the first viewing direction, the second AR position, and the second viewing direction.

[0119] Fig.10 The situation of the IoT device 30 to be located is schematically shown. Through AR glasses with known AR positions and viewing directions, the viewing direction from the first AR position 32 is determined as an angle α using a compass. Afterwards, a second viewing direction from the second AR position 34 toward the IoT device 30 to be located is determined. Therefore, through the first AR position 32, the second AR position 34, the first viewing direction α, and the second viewing direction β, the IoT position of the IoT device 30 can be determined using a compass. This position can be assigned to the IoT device and stored for future use. If the IoT device 30 is located indoors Fig.10 , the IoT device 30 can be used as an anchor point to determine the AR position and viewing direction of the AR glasses.

[0120] If compass information is not available, only the angle θ (the viewing angle between the two viewing positions) is known and a third AR position is needed to calculate the actual IoT position of the IoT device. The process is the same as the process described above for calculating the AR position of the AR glasses.

[0121] Fig.11 A device 110 is shown comprising a processor 112, wherein the processor 112 is connected to a memory 114. The memory 114 stores instructions which, when executed by the processor 112, implement the steps of the method as described above. In addition, the processor 112 is connected to a communication module 116 in order to receive tracking information, viewing direction, compass information or similar information from the AR glasses. The data transmission may be wirelessly transmitted via WLAN, WIFI, GSM, 3G, 4G, 5G, Bluetooth, ZigBee or any other standard.

[0122] Alternatively, instead of the communication module 116 , the processor 112 and the memory 114 of the device 110 are directly connected to the AR glasses as one.

[0123] Preferably, the processor may be any general purpose processor, ASIC, FPGA or any other kind of processing unit.

[0124] Preferably, the device is a terminal, such as a smart phone, a tablet, a laptop or any other kind of device. In particular, the device also includes a display screen for displaying an indoor map. The IoT device can then be displayed with its corresponding position on the indoor map. By interacting with the representation of the IoT device on the terminal, each IoT device in the home automation system can be controlled.

[0125] Therefore, through the present disclosure, real-time positioning and tracking of AR glasses worn by users on indoor maps are realized. Through AR glasses, IoT devices can be located. Through the precise IoT location of IoT devices, the control of IoT devices is simplified. In particular, an indoor map can be displayed on the terminal, indicating IoT devices on the indoor map, where the IoT location of each IoT device is determined according to the present disclosure. Then, IoT devices can be conveniently accessed through the location on the indoor map.

Claims

1. A method for locating an IoT device using augmented reality (AR) glasses, comprising: Get indoor maps; Determine, based on the initial information of the AR glasses, an AR position of the AR glasses on the indoor map and a viewing direction of the AR glasses; Receiving tracking information related to the IoT device to be located from the AR glasses; as well as Determine the IoT position of the IoT device to be located based on the determined AR position and viewing direction and in combination with the tracking information of the AR glasses; Wherein, based on the initial information of the AR glasses, determining the AR position of the AR glasses on the indoor map and the viewing direction of the AR glasses includes: Receiving tracking information related to at least three anchor points from the AR glasses, wherein the anchor point positions of the at least three anchor points on the indoor map are known; For any two anchor points of the at least three anchor points, determine the angle between the two anchor points according to respective tracking information of the two anchor points; and The AR position and the viewing direction of the AR glasses are determined according to the multiple angles.

2. The method according to claim 1, characterized in that: The initial information of the AR glasses includes: A user input indicating the AR position and the viewing direction of the AR glasses on the indoor map is received on the terminal.

3. The method according to claim 1, characterized in that Determining the angle between any two anchor points of the at least three anchor points includes: The direction / sign of an angle between the two anchor points is determined.

4. The method according to claim 1, characterized in that At least one anchor point includes an AR marker to be recognized by the AR glasses to identify the anchor point.

5. The method according to claim 1, characterized in that The AR glasses identify at least one anchor point through image recognition.

6. The method according to claim 1, characterized in that Determine compass information of the AR glasses, and orient the indoor map according to the compass information.

7. The method according to claim 6, characterized in that Based on the initial information of the AR glasses, determining the AR position of the AR glasses on the indoor map and the viewing direction of the AR glasses includes: The compass information is used to determine a plurality of angles between at least two anchor points and / or directions of angles between any two of the at least two anchor points.

8. The method according to any one of claims 1 to 7, characterized in that The anchor point is provided by one or more IoT devices.

9. The method according to any one of claims 1 to 7, characterized in that The indoor map is provided by a vacuum cleaner and / or a surveillance camera.

10. The method according to any one of claims 1 to 7, characterized in that Determining the IoT position of the IoT device to be located based on the determined position and viewing direction and in combination with the tracking information of the AR glasses includes: Determine a first viewing direction of the AR glasses from the first AR position to the IoT device to be located; Determining at least a second viewing direction of the AR glasses from a second AR position to the IoT device to be located; and The IoT position of the IoT device to be located on the indoor map is determined according to the first AR position, the first viewing direction, the second AR position, and the second viewing direction.

11. A method for locating an IoT device using augmented reality (AR) glasses, comprising: Get indoor maps; Determine, based on the initial information of the AR glasses, an AR position of the AR glasses on the indoor map and a viewing direction of the AR glasses; Receiving tracking information related to the IoT device to be located from the AR glasses; as well as Determine the IoT position of the IoT device to be located based on the determined AR position and viewing direction and in combination with the tracking information of the AR glasses; Wherein, based on the initial information of the AR glasses, determining the AR position of the AR glasses on the indoor map and the viewing direction of the AR glasses includes: Receiving tracking information about a first anchor point and at least one second anchor point from the AR glasses, wherein at least the first anchor point is moving and anchor point positions of the first anchor point and the at least one second anchor point on an indoor map are known; Determine a first angle between the first anchor point and the second anchor point at a first time point; determining a second angle between the first anchor point and the second anchor point at a second time point, wherein an anchor point position of the first anchor point at the second time point is different from an anchor point position of the first anchor point at the first time point; The AR position and the viewing direction of the AR glasses are determined according to the first angle and the second angle.

12. The method according to claim 11, characterized in that The initial information of the AR glasses includes: A user input indicating the AR position and the viewing direction of the AR glasses on the indoor map is received on the terminal.

13. The method according to claim 11, characterized in that Determining the angle between any two anchor points of the at least three anchor points includes: The direction / sign of an angle between the two anchor points is determined.

14. The method according to claim 11, characterized in that At least one anchor point includes an AR marker to be recognized by the AR glasses to identify the anchor point.

15. The method according to claim 11, characterized in that The AR glasses identify at least one anchor point through image recognition.

16. The method according to claim 11, characterized in that Determine compass information of the AR glasses, and orient the indoor map according to the compass information.

17. The method according to claim 16, characterized in that Based on the initial information of the AR glasses, determining the AR position of the AR glasses on the indoor map and the viewing direction of the AR glasses includes: The compass information is used to determine a plurality of angles between at least two anchor points and / or directions of angles between any two of the at least two anchor points.

18. The method according to any one of claims 11 to 17, characterized in that The anchor point is provided by one or more IoT devices.

19. The method according to any one of claims 11 to 17, characterized in that The indoor map is provided by a vacuum cleaner and / or a surveillance camera.

20. The method according to any one of claims 11 to 17, characterized in that Determining the IoT position of the IoT device to be located based on the determined position and viewing direction and in combination with the tracking information of the AR glasses includes: Determine a first viewing direction of the AR glasses from the first AR position to the IoT device to be located; Determining at least a second viewing direction of the AR glasses from a second AR position to the IoT device to be located; and The IoT position of the IoT device to be located on the indoor map is determined according to the first AR position, the first viewing direction, the second AR position, and the second viewing direction.

21. A data processing apparatus comprising a processor and a memory storing instructions adapted to perform a method according to any one of claims 1 to 10 or 11 to 20 when the instructions are executed by the processor.

22. A non-transitory computer-readable storage medium storing instructions for execution by a processor, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 10 or 11 to 20.

Images