Interactive device and method for controlling it

Abstract

A 1. An interactive device (2) comprising: - a frame member (4) arranged on a mobile robot (10), - a first display (6) and a second display (6') connected to the frame 5 member (4), wherein the displays (6, 6') are oriented in different di- rections, - at least one camera (8, 8') arranged to detect objects (14) in proximi- ty of the interactive device (2), - a control unit (16) communicatively connected to the displays (6, 6') 10 and the at least one camera (8, 8'), wherein the control unit (16) is configured to communicate with at first extern device (18) and a sec- ond extern device (18') is disclosed. The control unit (16) is configured to receive data (22) from at least one sensor of the interactive device (2) or the at least one camera (8, 15 8') and control the motion of the mobile robot (10) in dependency of said data (22).

Description

[0001] Interactive Device and Method for Controlling It

[0002] Field of invention

[0003] The present invention relates to an interactive device comprising:

[0004] - a frame member arranged on a mobile robot,

[0005] - a first display and a second display spaced apart from the first display connected to the frame member,

[0006] - at least one camera arranged to detect objects in proximity of the interactive device,

[0007] - a control unit communicatively connected to the displays and the at least one camera, wherein the control unit is configured to communicate with at first extern device and a second extern device.

[0008] The present invention moreover relates to a method for for controlling such interactive device.

[0009] Prior art

[0010] Interactive devices comprising one or more displays are widely used in airports and shopping centres. It is, however, difficult to ensure that such interactive devices are placed in the most optimum position in an airport or in a shopping centre.

[0011] WO 2016 / 065362 Al discloses a robot for providing customer service within a facility includes a locomotion platform. The robot comprises:

[0012] - an upper sensor for detecting objects within an upper field of view of the robot;

[0013] - a lower sensor for detecting objects within a lower field of view of the robot.

[0014] The robot computer is configured to detect the presence of a customer within the facility based on information received from at least one of the upper sensor and lower sensor, and the robot computer is further configured to access one or more databases storing information associated with products available to customers within the facility and to provide customer service to the customer based on the accessed information. Accordingly, the robot is capable of detecting an obstacle (by using one of its internal sensors) and take into account that the obstacle and thus moving the robot to a position that will avoid a collision between the robot and the obstacle. This enables the robot to take into considerations obstacle present in front of the robot. It is, however, not possible to take into consideration obstacles that are not located clos to the robot.

[0015] Thus, there is a need for an improved interactive device and an improved method for controlling the interactive device, by which it is possible to reduce or even eliminate the above mentioned disadvantages of the prior art.

[0016] Summary of the invention

[0017] The object of the present invention can be achieved by an interactive device as defined in claim 1 and by a methos as defined in claim 9. Preferred embodiments are defined in the dependent subclaims, explained in the following description and illustrated in the accompanying drawings.

[0018] The interactive device according to the invention is an interactive device comprising:

[0019] - a frame member arranged on a mobile robot,

[0020] - a first display and a second display connected to the frame member, wherein the displays are oriented in different directions,

[0021] - at least one camera arranged to detect objects in proximity of the interactive device,

[0022] - a control unit communicatively connected to the displays and the at least one camera, wherein the control unit is configured to communicate with at first extern device and a second extern device, wherein the control unit is configured to receive data from at least one sensor of the interactive device or the at least one camera and control the motion of the mobile robot in dependency of said data, wherein the control unit is used to receive external signals and use these signals for planning the motion of the mobile robot in dependency of the external signals.

[0023] Hereby, it is possible to ensure that the interactive device is placed in the most optimum position (e.g. an airport or in a shopping centre).

[0024] The frame member is arranged on a mobile robot. The frame member may have any suitable shape and size. In an embodiment, the frame member is designed as an elongated member configured to protrude in an upright direction from the mobile robot.

[0025] In an embodiment, the frame member is detachably attached to the mobile robot.

[0026] In an embodiment, the frame member is an integrated part of the mobile robot.

[0027] In an embodiment, the first display and the second display spaced apart from each other. In an embodiment, the first display and the second display are oriented in opposite directions to accommodate two users standing in front of each other.

[0028] The first display and the second display are connected to the frame member.

[0029] In an embodiment, the first display and the second display are detachably attached to the frame member.

[0030] In an embodiment, the first display is attached to a connection structure that is detachably attached to the frame member. In an embodiment, the second display is attached to a connection structure that is detachably attached to the frame member.

[0031] In an embodiment, the first display and the second display are integrated parts of the frame member.

[0032] In an embodiment, first display is spaced apart from the second display.

[0033] In an embodiment, the rear side of the first display faces the rear side of the second display.

[0034] The at least one camera arranged to detect objects in proximity of the interactive device. In an embodiment, the at least one camera arranged to detect objects in a range of 0-100 m of the interactive device. In an embodiment, the at least one camera arranged to detect objects in a range of 0-50 m of the interactive device. In an embodiment, the at least one camera arranged to detect objects in a range of 0-20 m of the interactive device.

[0035] The control unit communicatively connected to the displays and the at least one camera. In an embodiment, the control unit communicatively connected to the displays and the at least one camera by one or more wireless connections. In an embodiment, the control unit communicatively connected to the displays and the at least one camera by one or more wired connections.

[0036] The control unit is configured to communicate wireless with at first extern device and a second extern device.

[0037] In an embodiment, the control unit is configured to communicate wireless with at first extern device and a second extern device via the Internet. The control unit is configured to receive data from : a) at least one sensor of the interactive device or b) the at least one camera and control the motion of the mobile robot in dependency of said data. Hereby, the mobile robot can automatically be moved to a zone, in a desired predefined number of area specific persons are located.

[0038] In an embodiment, the control unit is configured to receive data from at least one sensor of the interactive device, wherein the at least one sensor includes a camera (e.g. a camera of the mobile robot).

[0039] In an embodiment, the sensor is a built-in sensor configured to identify the surroundings of the interactive device. Hereby, it is possible to plan and take the most efficient route and safely avoid obstacles and people.

[0040] In an embodiment, the at least one sensor includes a laser scanner (e.g. a laser scanner of the mobile robot).

[0041] In an embodiment, the at least one sensor includes an ultrasonic sensor (configured to measure the distance to an object using ultrasonic sound waves), such ultrasonic sensor would typically use a transducer to send and receive ultrasonic pulses that relay back information about an object's proximity. laser scanner (e.g. a laser scanner of the mobile robot).

[0042] It is an advantage that the control unit is configured to receive external signals and use these signals for planning the motion of the mobile robot in dependency of the external signals.

[0043] It is advantageous that the external signals include data from one or more external sensors (e.g. one or more sensors placed in an airport or in a shopping centre). The one or more sensors may include one or more cameras, one or more proximity sensors, one or more "light detection and ranging" (LADAR.) techniques (a method for determining ranges by targeting an object or a surface with a laser and measuring the time for the reflected light to return to the receiver) e.g. LIDAR scanning or 3D laser scanning).

[0044] In an embodiment, the control unit is configured to process at least one two-user computer game involving a first user applying the first extern device and a second user applying the second extern device in a manner, in which: a) the first display provides visual guidance and feedback for the first user and b) the second display provides visual guidance and feedback for the second user.

[0045] In an embodiment, the first and second extern device is a smartphone.

[0046] In an embodiment, the first and second extern device is a tablet.

[0047] The mobile robot typically comprises several ground-engaging wheels.

[0048] In an embodiment, the control unit is configured to calculate a route in dependency of the data and control the motion of the mobile robot along the route in dependency of said data, wherein the route comprises a plurality of zones.

[0049] The data may comprise information provided by one or more internal sensors. The data may comprise information provided by one or more external sensors.

[0050] In an embodiment, the control unit is configured to determine the area specific number (the number of people per area unit) of people in each zone and control the motion of the mobile robot in dependency of the determined area specific number of people in the zones.

[0051] Hereby, it is possible to automatically ensure that the interactive device is placed in a zone having a predefined area specific number of people. Accordingly, it can be avoided that too few people or too many people are present in the zone, in which the interactive device is placed.

[0052] If too many people are present in a zone, it may be a problem to move the interactive device through the zone because the zone is too crowded.

[0053] On the other hand, if too few people are present in a zone, the interactive device is not of any use. Accordingly, it would be desirable to automatically move the interactive device to another zone having an area specific number of people in a predefined range.

[0054] In an embodiment, the control unit is configured to identify human faces from digital images or video frames captured by the at least one camera and determine the area specific number of people in each zone on the basis of the number of human faces.

[0055] In an embodiment, the control unit is configured to identify one or more human faces from digital images or video frames captured by the at least one camera and create a real-time avatar facial animation on the basis of the one or more human faces.

[0056] Avatar animation is created using a process called motion capture. This process uses real-time data captured from facial recognition techniques

[0057] In an embodiment, the control unit is configured to control the device and ensure that the device is maintained in a predefined zone in a time period determined in dependency of detected data.

[0058] To increase the comfort of people it is possible to use the safety system of the mobile robot and add an additional safety distance (e.g. 20 cm) to the mobile robot to avoid that the device gets too close to people.

[0059] In an embodiment, the control unit is configured to add a pause before the interactive device is moved from one zone to another. In an embodiment, the pause is in the range 1-120 seconds. In an embodiment, the pause is in the range 2-60 seconds. In an embodiment, the pause is in the range 4-30 seconds. In an embodiment, the pause is in the range 5-20 seconds.

[0060] The method according to the invention is a method for controlling an interactive device comprising:

[0061] - a frame member arranged on a mobile robot,

[0062] - a first display and a second display connected to the frame member,

[0063] - at least one camera arranged to detect objects in proximity of the interactive device,

[0064] - a control unit communicatively connected to the displays and the at least one camera, wherein the control unit is configured to communicate with at first extern device and a second extern device, wherein the method comprises the step of applying the control unit to receive data from : a) at least one sensor of the interactive device or b) the at least one camera and control the motion of the mobile robot in dependency of said data, wherein the control unit is used to receive external signals and use these signals for planning the motion of the mobile robot in dependency of the external signals, wherein the external signals include data from one or more external sensors. In an embodiment, the control unit is used to process at least one two- user computer game involving a first user applying the first extern device and a second user applying the second extern device in a manner, in which: a) the first display provides visual guidance and feedback for the first user and b) the second display provides visual guidance and feedback for the second user, wherein the control unit is configured to receive data from the at least one camera and control the motion of the mobile robot in dependency of said data.

[0065] In an embodiment, the control unit is used to calculate a route in dependency of the data and control the motion of the mobile robot along the route in dependency of said data, wherein the route comprises a plurality of zones.

[0066] In an embodiment, the control unit is used to determine the area specific number of people in each zone and control the motion of the mobile robot in dependency of the determined area specific number of people in the zones.

[0067] In an embodiment, the control unit is used to identify human faces from digital images or video frames captured by the at least one camera and determine the area specific number of people in each zone on the basis of the number of human faces.

[0068] In an embodiment, the control unit is used to identify one or more human faces from digital images or video frames captured by the at least one camera and create a real-time avatar facial animation on the basis of the one or more human faces. In an embodiment, the control unit is used to control the device and ensure that the device is maintained in a predefined zone in a time period determined in dependency of detected data.

[0069] In an embodiment, the control unit is used to receive external signals and use these signals for planning the motion of the mobile robot in dependency of the external signals.

[0070] Description of the Drawings

[0071] The invention will become more fully understood from the detailed description given herein below. The accompanying drawings are given by way of illustration only, and thus, they are not limitative of the present invention. In the accompanying drawings:

[0072] Fig. 1A shows a schematic side view of an interactive device according to the invention in a configuration, in which a single user is using the interactive device;

[0073] Fig. IB shows a schematic side view of an interactive device according to the invention in a configuration, in which two users are using the interactive device;

[0074] Fig. 2 shows another schematic view of an interactive device according to the invention;

[0075] Fig. 3 shows a schematic top view of a building comprising a plurality of zones;

[0076] Fig. 4 shows a schematic side view of an interactive device according to the invention;

[0077] Fig. 5 shows a front view of the interactive device shown in Fig. 4 and

[0078] Fig. 6 shows a front view of the interactive device shown in Fig. 4 and Fig. 5.

[0079] Detailed description of the invention

[0080] Referring now in detail to the drawings for the purpose of illustrating preferred embodiments of the present invention, an interactive device 2 of the present invention is illustrated in Fig. 1A.

[0081] Fig. 1A illustrates a schematic side view of an interactive device 2 according to the invention in a configuration, in which a single user 20 is using the interactive device 2.

[0082] The interactive device 2 comprises a mobile robot 10 and a frame member 4 arranged on the mobile robot 10. The interactive device 2 comprises a first display 6 and a second display 6' spaced apart from the first display 6. The displays 6, 6' are connected to the frame member 4.

[0083] In an embodiment, the displays 6, 6' are integrated into the frame member 4. In an embodiment, the frame member 4 is integrated into the mobile robot 10.

[0084] The interactive device 2 comprises at least one camera 8, 8' arranged to detect objects in proximity of the interactive device 2.

[0085] In an embodiment, the interactive device 2 comprises a single camera 8, 8' arranged to detect objects in proximity of the interactive device 2.

[0086] In an embodiment, the interactive device 2 comprises two cameras 8, 8' arranged to detect objects in proximity of the interactive device 2.

[0087] In an embodiment, the interactive device 2 comprises three cameras 8, 8' arranged to detect objects in proximity of the interactive device 2.

[0088] In an embodiment, the interactive device 2 comprises more than three cameras 8, 8' arranged to detect objects in proximity of the interactive device 2. The interactive device 2 comprises a control unit 16 that is communicatively connected to the displays 6, 6' and the at least one camera 8, 8'.

[0089] The control unit 16 is configured to communicate with a first extern device 18 and a second extern device (not shown).

[0090] In an embodiment, the control unit 16 is integrated in the mobile robot 10. A mobile robot 10 typically comprises a control unit 16.

[0091] The control unit 16 is configured to receive data 22 from the at least one camera 8, 8' and control the motion of the mobile robot 10 in dependency of said data 22. The mobile robot 10 comprises groundengaging wheels 12, 12'.

[0092] The user 20 is using an extern device 18 to communicate with the interactive device 2. The extern device 18 is a smartphone. In an embodiment, the extern device 18 is a tablet. In an embodiment, the control unit 16 is configured to allow the user 20 to access information by using the extern device 18. The information may be available on either of the displays 6, 6'.

[0093] In an embodiment, the control unit 16 is configured to allow the user 20 to play a computer game by using the extern device 18.

[0094] Fig. IB illustrates a schematic side view of an interactive device 2 according to the invention in a configuration, in which two users 20, 20' are using the interactive device 2. The interactive device 2 corresponds to the one shown in Fig. 1A.

[0095] The first user 20 is using a first extern device 18. The second user 20' is using a second extern device 18'. In an embodiment, the users 20, 20 are playing a two-player computer game by using the interactive device 2 and the extern devices 18, 18'. Fig. 2 illustrates another schematic view of an interactive device 2 according to the invention. The interactive device 2 corresponds to the one shown in Fig. 1A and in Fig. IB.

[0096] The interactive device 2 is applying the cameras 8, 8' to determine the number of area specific persons are located in the zone in which the interactive device 2 is located. If the number of area specific persons is within a predefined acceptable range (e.g. 0.05-0.25 people per square meter), the interactive device 2 will maintain its position.

[0097] If, however, the number of area specific persons is not within a predefined acceptable range, the interactive device 2 will calculate a route and move to another zone. In this process it may be advantageous to apply external data (e.g. data from sensors located in other location of the building, in which the interactive device is located). Such sensors may include one or more cameras and / or scanners.

[0098] The external data may also include historical data and / or forecast data.

[0099] Hereby, the mobile robot can automatically be moved to a zone, in a desired predefined number of area specific persons are located.

[0100] The cameras 8, 8' of the interactive device 2 can detect objects 14 such as people or obstacles (e.g. a pillar, a plant or a vending machine).

[0101] The cameras 8, 8' will detect that two users 20, 20' are in close proximity of the interactive device 2. Accordingly, the control unit of the interactive device 2 will detect that two potential players have approached the interactive device 2. The interactive device 2 may generate a visual or audible signal indicating that the uses 20, 20' can play a two-player computer game on the interactive device 2 by using external devices (e.g. smartphones). The users 20, 20 can play as indicated and ex- plained with reference to Fig. 1A and Fig. IB.

[0102] Fig. 3 illustrates a schematic top view of a building 26 comprising a plurality of zones A, B, C, D, E, F, G, H. A plurality of sensors 28, 28' are located in the building 26. An interactive device 2 corresponding to the one shown in Fig. 1A, Fig. IB and in Fig. 2 is illustrated in Fig. 3.

[0103] A first sensor 28 wirelessly transmits information containing data 22 to the control unit of the interactive device 2. A second sensor 28' wirelessly transmits information containing data 22' to the control unit of the interactive device 2. In an embodiment, the control unit of the interactive device 2 is configured to determine the area specific number (the number of people per area unit) of people in zone G and in zone E. Accordingly, the control unit can control the motion of the mobile robot in dependency of the determined area specific number of people in zone E and zone G. All though it is not indicated in Fig. 3, the control unit of the interactive device 2 is configured to determine the area specific number of people in the other zones on the basis of data received for sensors located in these zones. Accordingly, the interactive device 2 is configured to control the mobile robot and move the interactive device 2 to another zone if the area specific number of people is no longer in a predefined range.

[0104] Fig. 4 illustrates a schematic side view of an interactive device 2 according to the invention. The interactive device 2 comprises a mobile robot 10 and a frame member 4 arranged on the mobile robot 10. The mobile robot 10 comprises ground-engaging wheels 12, 12'.

[0105] The interactive device 2 comprises a first display 6 and a second display 6' spaced apart from the first display 6. The displays 6, 6' are connected to the frame member 4.

[0106] The interactive device 2 comprises a first camera 8 connected to the first display 6. The interactive device 2 comprises a second camera 8' connected to the second display 6'.

[0107] The interactive device 2 comprises a control unit (not shown) that is communicatively connected to the displays 6, 6' and the cameras 8, 8'.

[0108] The interactive device 2 comprises a stop knob 32, by which it is possible to stop the interactive device 2.

[0109] Fig. 5 illustrates a front view of the interactive device 2 shown in Fig. 4. The interactive device 2 comprises a sensor 30 located at the frame member 4. The sensor 30 may be a camera or a laser scanner by way of example.

[0110] A first camera 8 is connected to the first display 6. The mobile robot 10 may comprise one or more sensors including one or more laser scanners, one or more ultrasonic sensors and one or more cameras.

[0111] Fig. 6 illustrates a front view of the interactive device shown in Fig. 4 and Fig. 5.

[0112] List of reference numerals

[0113] 2 Interactive device

[0114] 4 Frame member

[0115] 6, 6' Display

[0116] 8, 8' Camera

[0117] 10 Mobile robot

[0118] 12 Wheel

[0119] 14 Object

[0120] 16 Control unit

[0121] 18, 18' Extern device

[0122] 20, 20' User

[0123] 22, 22' Data

[0124] 24 Route

[0125] 26 Building

[0126] 28 Sensor

[0127] 30 Sensor

[0128] 32 Stop knob

[0129] A, B, C Zone

[0130] D, E, F Zone

[0131] G, H Zone

Claims

Claims1. An interactive device (2) comprising:- a frame member (4) arranged on a mobile robot (10),- a first display (6) and a second display (6') connected to the frame member (4), wherein the displays (6, 6') are oriented in different directions,- at least one camera (8, 8') arranged to detect objects (14) in proximity of the interactive device (2),- a control unit (16) communicatively connected to the displays (6, 6') and the at least one camera (8, 8'), wherein the control unit (16) is configured to communicate with at first extern device (18) and a second extern device (18'), wherein the control unit (16) is configured to receive data (22) from the at least one sensor of the interactive device (2) or camera (8, 8') and control the motion of the mobile robot (10) in dependency of said data (22), characterised in that the control unit (16) is used to receive external signals and use these signals for planning the motion of the mobile robot (10) in dependency of the external signals, wherein the external signals include data from one or more external sensors.

2. An interactive device (2) according to claim 1, wherein the control unit (16) is configured to process at least one two-player computer game involving a first user (20) applying the first extern device (18) and a second user (20') applying the second extern device (18') in a manner, in which: a) the first display (6) provides visual guidance and feedback for the first user (20) and b) the second display (6) provides visual guidance and feedback for the second user (20).

3. An interactive device (2) according to one of the preceding claims, wherein the control unit (16) is configured to calculate a route (24) in dependency of the data (22) and control the motion of the mobile robot (10) along the route (24) in dependency of said data (22), wherein the route (24) comprises a plurality of zones (A, B, C, D, E, F, G, H).

4. An interactive device (2) according to one of the preceding claims, wherein the control unit (16) is configured to determine the area specific number of people in each zone (A, B, C, D, E, F, G, H) and control the motion of the mobile robot (10) in dependency of the determined area specific number of people in the zones (A, B, C, D, E, F, G, H).

5. An interactive device (2) according to one of the preceding claims, wherein the control unit (16) is configured to identify human faces from digital images or video frames captured by the at least one camera (8, 8') and determine the area specific number of people in each zone (A, B, C, D, E, F, G, H) on the basis of the number of human faces.

6. An interactive device (2) according to one of the preceding claims 1- 4, wherein the control unit (16) is configured to identify one or more human faces from digital images or video frames captured by the at least one camera (8, 8') and create a real-time avatar facial animation on the basis of the one or more human faces.

7. An interactive device (2) according to one of the preceding claims, wherein the control unit (16) is configured to control the device (2) and ensure that the device (2) is maintained in a predefined zone (A, B, C, D, E, F, G, H) in a time period determined in dependency of detected data (22).

8. System comprising:- an interactive device (2) according to one of the preceding claims and- a building (26) comprising a plurality of zones (A, B, C, D, E, F, G, H) and a plurality of sensors (28, 28') located in the building (26).

9. Method for controlling an interactive device (2) comprising:- a frame member (4) arranged on a mobile robot (10),- a first display (6) and a second display (6') connected to the frame member (4),- at least one camera (8, 8') arranged to detect objects (14) in proximity of the interactive device (2),- a control unit (16) communicatively connected to the displays (6, 6') and the at least one camera (8, 8'), wherein the control unit (16) is configured to communicate with at first extern device (18) and a second extern device (18'), wherein the method comprises the step of applying the control unit (16) to receive data (22) from at least one sensor of the interactive device (2) or the at least one camera (8, 8') and control the motion of the mobile robot (10) in dependency of said data (22), characterised in that the control unit (16) is used to receive external signals and use these signals for planning the motion of the mobile robot (10) in dependency of the external signals, wherein the external signals include data from one or more external sensors.

10. A method according to claim 9, wherein the control unit (16) is used to process at least one two-user computer game involving a first user (20) applying the first extern device (18) and a second user (20') applying the second extern device (18') in a manner, in which: a) the first display (6) provides visual guidance and feedback for the first user (20) and b) the second display (6) provides visual guidance and feedback for the second user (20), wherein the control unit (16) is configured to receive data (22) from theat least one camera (8, 8') and control the motion of the mobile robot (10) in dependency of said data (22).

11. A method according to claim 9 or 10, wherein the control unit (16) is used to calculate a route (24) in dependency of the data (22) and control the motion of the mobile robot (10) along the route (24) in dependency of said data (22), wherein the route (24) comprises a plurality of zones (A, B, C, D, E, F, G, H).

12. A method according to one of the claims 9-11, wherein the control unit (16) is used to determine the area specific number of people in each zone (A, B, C, D, E, F, G, H) and control the motion of the mobile robot (10) in dependency of the determined area specific number of people in the zones (A, B, C, D, E, F, G, H).

13. A method according to one of the preceding claims 9-12, wherein the control unit (16) is used to identify human faces from digital images or video frames captured by the at least one camera (8, 8') and determine the area specific number of people in each zone (A, B, C, D, E, F, G, H) on the basis of the number of human faces.

14. A method according to one of the preceding claims 9-12, wherein the control unit (16) is used to identify one or more human faces from digital images or video frames captured by the at least one camera (8, 8') and create a real-time avatar facial animation on the basis of the one or more human faces.

15. A method according to one of the preceding claims 9-14, wherein the control unit (16) is used to control the device (2) and ensure that the device (2) is maintained in a predefined zone (A, B, C, D, E, F, G, H) in a time period determined in dependency of detected data (22).

Images