Control device and robot

Abstract

To set a photographed image of a first terminal device that is imaged for a remote operation by a robot performing a first touch operation on the first terminal device according to an application executed by the remote operation.SOLUTION: A control device 30A communicating with a robot and a second terminal device comprises a control unit 300a1, an acquisition unit 300a2, a specification unit 300a6, and an update unit 300a7. The acquisition unit 300a2 acquires a photographed image and second position information indicating a touch position with respect to the second terminal device. The control unit 300a1 causes the second terminal device to display a second icon corresponding to a first icon displayed on the first terminal device and causes the robot to execute the first touch operation on the basis of the second position information. The specification unit 300a6 specifies the first icon corresponding to the second icon indicating the second position information. The update unit 300a7 updates a frame rate of the photographed image on the basis of the type of an application corresponding to the specified first icon.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a control device and a robot. [Background technology]

[0002] Patent Document 1 discloses an inspection device that inspects the functionality of a terminal device such as a smartphone using a robot that executes touch operations on the terminal device. The inspection device disclosed in Patent Document 1 inspects the functionality of the terminal device by having the robot execute touch operations on the terminal device to be inspected and comparing an image obtained by capturing an image of the display unit of the terminal device with a predetermined gas image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-290852 Summary of the Invention [Problem to be solved by the invention]

[0004] When a terminal device is remotely controlled using a robot to execute an application, the user displays a captured image of the display area of ​​the operated terminal device on the terminal device at hand and understands the execution status of the application through the image displayed on the terminal device at hand. When remotely controlling a terminal device using a robot, various problems can arise if the image capture settings of the terminal device running an application are not suitable for the application. A specific example of the image capture settings is the resolution of the captured image. When the captured image is a moving image, a specific example of the image capture settings is the frame rate of the captured image. For example, if the frame rate of the captured image on a terminal device running a video viewing application or a game application is low, it can interfere with video viewing or game operation on the operating terminal device. On the other hand, if the frame rate of the captured image is set high in advance regardless of the type of application, when the application being remotely executed is an email application or a social networking application, a low frame rate of the captured image does not interfere with remote control, but the amount of communication traffic related to the captured image unnecessarily increases.

[0005] The present disclosure has been made in consideration of the problems described above, and aims to enable a terminal device to be remotely controlled using a robot, with an image of the display area of ​​the terminal device captured using image capture settings that correspond to the application to be executed by the terminal device. [Means for solving the problem]

[0006] In order to solve the above-described problems, a control device according to the present disclosure controls a robot that performs a first touch operation on a display area of ​​a first terminal device, in which a plurality of first icons corresponding one-to-one to a plurality of applications are displayed, based on a second touch operation on a display area of ​​a second terminal device. The control device includes an acquisition unit, a control unit, an identification unit, and an update unit. The acquisition unit acquires a first image generated by the robot capturing an image of the first terminal device and second position information indicating a position in the display area of ​​the second terminal device touched by the second touch operation. The control unit displays a plurality of second icons corresponding one-to-one to the plurality of first icons on the display area of ​​the second terminal device based on the first image, and causes the robot to perform the first touch operation based on the second position information. The identification unit identifies a first icon from the plurality of second icons that corresponds to a second icon selected by the second touch operation based on the second position information. The update unit updates settings for imaging by the robot to settings according to the type of application corresponding to the first icon, based on the type of application corresponding to the first icon.

[0007] In order to solve the above-described problems, another aspect of the present disclosure provides a control device that controls a robot that performs a first touch operation on a display area of ​​a first terminal device, where a plurality of first icons corresponding one-to-one to a plurality of applications are displayed, based on a second touch operation on a display area of ​​a second terminal device. The control device includes an acquisition unit, a control unit, an identification unit, and an update unit. The acquisition unit acquires a first image generated by the robot capturing an image of the first terminal device and second position information indicating a position in the display area of ​​the second terminal device that was touched by the second touch operation. The control unit displays a plurality of second icons corresponding one-to-one to the plurality of first icons on the display area of ​​the second terminal device based on the first image, and causes the robot to perform the first touch operation based on the second position information. The identification unit identifies one of the plurality of first icons selected by the first touch operation based on the first image. The update unit updates settings for imaging by the robot to settings according to the type of application corresponding to the one first icon, based on the type of application corresponding to the one first icon.

[0008] In order to solve the above-described problems, a robot of the present disclosure is a robot that performs a first touch operation on a display area of ​​a first terminal device based on a second touch operation on a display area of ​​a second terminal device, The system includes an imaging device, an operation device, and a processing device. The operation device is located between the first terminal device and the imaging device and performs the first touch operation. The processing device includes an acquisition unit, a control unit, and an update unit. The acquisition unit acquires instruction information related to the first touch operation and setting information related to imaging settings of the imaging device. The control unit causes the operation device to perform the first touch operation based on the instruction information. The update unit updates the setting of the imaging device based on the setting information.

[0009] In order to solve the above problems, another aspect of the present disclosure provides a robot that performs a first touch operation on a display area of ​​a first terminal device, on which a plurality of first icons corresponding one-to-one to a plurality of applications are displayed, based on a second touch operation on a display area of ​​a second terminal device, and includes an imaging device, an operation device, and a processing device. The operation device is located between the first terminal device and the imaging device and performs the first touch operation. The processing device includes an acquisition unit, a control unit, an identification unit, and an update unit. The acquisition unit acquires instruction information related to the first touch operation. The control unit causes the operation device to perform the first touch operation based on the instruction information. The identification unit identifies one of the plurality of first icons selected by the first touch operation based on a first image showing the display area of ​​the first terminal device captured by the imaging device. The update unit updates settings for imaging by the imaging device to settings according to the type of application corresponding to the one first icon, based on the type of application corresponding to the one first icon. [Effects of the Invention]

[0010] According to the present disclosure, when a terminal device is remotely controlled using a robot, the imaging settings when the robot captures an image of the display area of ​​the terminal device can be set to settings that correspond to the application that is to be executed by the terminal device via remote control. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating a configuration example of a remote control system 1A according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram showing a configuration example of a robot 10A. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a terminal device 20A. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a control device 30A. [Figure 5] 10A and 10B are diagrams for explaining the inclination of the terminal device 40 relative to the robot 10A. [Figure 6]10 is a flowchart showing the flow of a display operation for displaying an image in the display area of ​​the terminal device 40 on the display device 230 of the terminal device 20A. [Figure 7] 10 is a diagram showing an example of an image captured by an imaging device 130 of a terminal device 40. FIG. [Figure 8] FIG. 10 is a diagram showing an example of a terminal image generated by a control device 30A. [Figure 9] 10 is a flowchart showing a flow of a remote control operation for making the robot 10A execute a touch operation on the terminal device 40 in response to a touch operation performed on the display device 230 of the terminal device 20A. [Figure 10] FIG. 10 is a diagram illustrating a configuration example of a remote control system 1B according to a second embodiment of the present disclosure. [Figure 11] FIG. 2 is a diagram illustrating an example of the configuration of a terminal device 20B. [Figure 12] 10 is a flowchart showing the flow of a display operation for displaying an image in the display area of ​​the terminal device 40 on the display device 230 of the terminal device 20B. [Figure 13] 10 is a flowchart showing a flow of a remote control operation for making the robot 10A execute a touch operation on the terminal device 40 in response to a touch operation performed on the display device 230 of the terminal device 20B. [Figure 14] FIG. 10 is a diagram illustrating a configuration example of a remote control system 1C according to a third embodiment of the present disclosure. [Figure 15] FIG. 2 is a diagram illustrating an example of the configuration of a robot 10C. [Figure 16] 10 is a flowchart showing the flow of a display operation for displaying an image in the display area of ​​the terminal device 40 on the display device 230 of the terminal device 20A. [Figure 17] 10 is a flowchart showing a flow of a remote control operation in which the robot 10C executes a touch operation on the terminal device 40 in response to a touch operation performed on the display device 230 of the terminal device 20A. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Various technically preferable limitations are applied to the embodiments described below. However, the embodiments of the present disclosure are not limited to the forms described below.

[0013] A. First embodiment FIG. 1 is a diagram illustrating a configuration example of a remote control system 1A according to a first embodiment of the present disclosure. As shown in FIG. 1, the remote control system 1A includes a robot 10A and a control device 30A. The robot 10A communicates with the control device 30A via a communication network NW. The communication network NW includes the Internet and a mobile communication network. The remote control system 1A is a system for remotely controlling a terminal device 40 using a terminal device 20A via the communication network NW. The terminal device 20A communicates with the control device 30A via the communication network NW. In this embodiment, the control device 30A controls the robot 10A in response to a user's operation on the terminal device 20A, thereby achieving remote control of the terminal device 40. The terminal device 40 is an example of a first terminal device in the present disclosure. The terminal device 20A is an example of a second terminal device in the present disclosure.

[0014] The terminal device 40 in FIG. 1 is an information processing device equipped with a program execution function. The terminal device 40 is, for example, a smartphone. Although detailed illustration is omitted in FIG. 1, the planar shape of the housing of the terminal device 40 is rectangular. The planar shape of the housing of the terminal device 40 may be a polygon other than a rectangle, or a shape formed by at least a portion of a curve, such as a rectangle with four rounded corners. Multiple application programs are pre-installed in the terminal device 40. The application programs may also be referred to as "application software" or "apps." The terminal device 40 displays multiple icons in a display area, each of which corresponds one-to-one to multiple applications. The icon displayed in the display area of ​​the terminal device 40 is an example of a first icon in the present disclosure. The terminal device 40 executes an application corresponding to an icon designated by a touch operation on the display area.

[0015] Fig. 2 is a diagram showing an example of the configuration of the robot 10A. As shown in Fig. 2, the robot 10A includes a processing device 100, a communication device 110, a storage device 120, an imaging device 130, and an operation device 140. The communication device 110 is a wireless communication module. The communication device 110 performs wireless communication with a base station of a communication network NW.

[0016] The storage device 120 is a recording medium readable by the processing device 100. The storage device 120 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM).

[0017] A program 120a1 to be executed by the processing device 100 is stored in the nonvolatile memory of the storage device 120. The volatile memory of the storage device 120 is used by the processing device 100 as a work area when executing the program 120a1. The program 120a1 is obtained, for example, from a server (not shown) connected to the communication network NW via the communication device 110, and then stored in the storage device 120. The program 120a1 may be pre-stored in the storage device 120 when the robot 10A is shipped from a factory.

[0018] The imaging device 130 is, for example, a camera equipped with a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), which is an imaging element that converts collected light into an electrical signal. In this embodiment, the imaging device 130 captures a moving image including the terminal device 40. The imaging device 130 outputs image data representing the captured image to the processing device 100.

[0019] As shown in FIG. 1, the operating device 140 includes an indicator 142 for performing a touch operation on the terminal device 40, and an arm 144. The operating device 140 also includes a movement mechanism (not shown) for moving the arm 144. When performing a touch operation using the indicator 142, the movement mechanism moves the arm 144 downward. The indicator 142 is formed in a rod shape. Although not shown in detail in FIG. 1, one end of the indicator 142 is provided with a contact portion for pressing a display area of ​​the terminal device 40. The other end of the indicator 142 is provided with a light-emitting portion that emits light when the contact portion is in contact with another object (for example, the terminal device 40). The arm 144 moves the indicator 142 between the imaging device 130 and the terminal device 40. The arm 144 is made of a transparent material such as transparent resin so as not to interfere with the imaging device 130 capturing an image of the terminal device 40. The operation device 140 may have multiple pairs of indicators 142 and arms 144. An operation device 140 having two pairs of indicators 142 and arms 144 can perform pinch operations such as pinch-in or pinch-out on the terminal device 40. Pinch-in refers to an operation of touching two fingers on the display area of ​​the terminal device 40 and moving the two fingers so as to narrow the distance between the touch position of one finger and the touch position of the other finger. Pinch-in is generally used as an operation to instruct the terminal device 40 to reduce the size of a displayed image. Pinch-out refers to an operation of touching two fingers on the display area of ​​the terminal device 40 and moving the two fingers so as to widen the distance between the touch position of one finger and the touch position of the other finger. Pinch-out is generally used as an operation to instruct the terminal device 40 to enlarge a displayed image.

[0020] The processing device 100 includes a processor, such as a CPU (Central Processing Unit), i.e., a computer. The processing device 100 may be configured as a single computer or multiple computers. When the robot 10A is powered on, the processing device 100 reads the program 120a1 from the non-volatile memory to the volatile memory and starts executing the program 120a1. The processing device 100 operating in accordance with the program 120a1 functions as the control unit 100a1 shown in FIG. 2. The control unit 100a1 shown in FIG. 2 is a software module realized by operating the processing device 100 in accordance with the program 120a1.

[0021] The control unit 100a1 causes the imaging device 130 to capture an image of the terminal device 40. The imaging device 130 generates a captured image by capturing an image of the terminal device 40. The captured image is an example of a first image in the present disclosure. The control unit 100a1 transmits the captured image of the terminal device 40 captured by the imaging device 130 to the communication network NW via the communication device 110. The control unit 100a1 also updates the settings of the imaging device 130 in response to an update instruction provided from the communication network NW. The settings of the imaging device 130 include at least one of the frame rate and the resolution of the captured image. In this embodiment, the setting of the imaging device 130 is the frame rate of the captured image. The update instruction includes setting information indicating the settings of the imaging device 130.

[0022] The control unit 100a1 controls the operation device 140 in response to an operation instruction received from the communication network NW, and performs a touch operation on the display area of ​​the terminal device 40. The touch operation performed by the robot 10A is an example of a first touch operation in the present disclosure. In the present embodiment, an operation instruction specifying a touch position in a two-dimensional coordinate system with the center of the robot 10A as the origin is given to the robot 10A from the control device 30A. The operation instruction is an example of instruction information in the present disclosure.

[0023] Next, the configuration of the terminal device 20A will be described. The terminal device 20A is an information processing device that accepts user operations for remotely operating the terminal device 40. The terminal device 20A is, for example, a smartphone. FIG. 3 is a diagram showing an example configuration of the terminal device 20A. As shown in FIG. 3, the terminal device 20A includes a processing device 200, a communication device 210, a storage device 220, a display device 230, and an input device 240. Like the processing device 100, the processing device 200 is configured from a single computer or multiple computers. Like the communication device 110, the communication device 210 is a wireless communication module. The communication device 210 performs wireless communication with a base station of the communication network NW. Like the storage device 120, the storage device 220 includes a non-volatile memory and a volatile memory.

[0024] A program 220a1 to be executed by the processing device 200 is stored in the nonvolatile memory of the storage device 220. The volatile memory of the storage device 220 is used by the processing device 200 as a work area when executing the program 220a1. The program 220a1 is obtained, for example, from a server (not shown) connected to the communication network NW via the communication device 210, and then stored in the storage device 220. The program 220a1 may be pre-stored in the storage device 220 at the time of shipping the terminal device 20A from a factory.

[0025] The display device 230 includes, for example, a liquid crystal display and a drive circuit for the liquid crystal display. The display device 230 displays various images under the control of the processing device 200. The display device 230 is an example of a display unit of the second terminal device in the present disclosure. The input device 240 is, for example, a transparent sheet-like contact detection sensor. The input device 240 is provided so as to cover the display area of ​​the display device 230. In this embodiment, the display device 230 and the input device 240 form a touch panel. When a user performs a touch operation on the input device 240, the input device 240 outputs operation data indicating the user's operation to the processing device 200. The operation data is data indicating the user's touch operation. The operation data includes data indicating the user's touch position, i.e., the touch position within the display area of ​​the display device 230. By outputting the operation data indicating the touch operation to the processing device 200, the touch position within the display area of ​​the display device 230 is transmitted to the processing device 200. When a long press operation in which the finger is continuously pressed on the touch position and then released is performed on the input device 240, the input device 240 outputs operation data further including data indicating the touch time to the processing device 200. When an operation in which the finger is moved while touching and then released, such as a flick operation, is performed on the input device 240, the input device 240 outputs operation data including data indicating each of the touch start position, touch end position, and touch time to the processing device 200.

[0026] Upon receiving operation data indicating an operation instructing the start of execution of program 220a1 from input device 240, processing device 200 reads program 220a1 from nonvolatile memory to volatile memory and starts execution of program 220a1. Program 220a1 may be executed by being called from a program other than program 220a1. When processing device 200 is capable of executing multiple programs in parallel, if the currently executing program contains code instructing the call of program 220a1, processing device 200 starts execution of program 220a1 upon execution of the code. A specific example of an operation instructing the start of execution of program 220a1 is a touch operation on an icon displayed on display device 230 as representing program 220a1. Processing device 200 operating in accordance with program 220a1 functions as control unit 200a1. Control unit 200a1 shown in FIG. 3 is a software module realized by causing processing device 200 to operate in accordance with program 220a1.

[0027] The control unit 200a1 receives a terminal image transmitted from the control device 30A via the communication device 210. The terminal image refers to an image of the display area of ​​the terminal device 40. The terminal image is generated by the control device 30A from a captured image of the terminal device 40. The control unit 200a1 also displays the received terminal image on the display device 230, thereby prompting the user to perform a touch operation. The control unit 200a1 then transmits position information indicating the position of a touch operation performed on the display area of ​​the display device 230 to the control device 30A via the communication device 210. Because the terminal image is displayed in the display area of ​​the display device 230, a position within the terminal image is specified by a touch operation performed on the display area of ​​the display device 230. The touch operation performed on the display area of ​​the display device 230 is an example of a second touch operation in the present disclosure. The position information transmitted from the terminal device 20A to the control device 30A is an example of second position information in the present disclosure.

[0028] Next, the configuration of the control device 30A will be described. The control device 30A is an information processing device that controls the robot 10A in response to a user's touch operation on the terminal device 20A. The control device 30A is, for example, a server or a personal computer. FIG. 4 is a diagram showing an example configuration of the control device 30A. As shown in FIG. 4, the control device 30A includes a processing device 300, a communication device 310, and a storage device 320. The processing device 300, like the processing device 100, is configured from a single computer or multiple computers. The communication device 310 is a wired communication module. The communication device 310 is connected to the communication network NW via a communication line. The communication device 310 performs wired communication with the communication network NW. Like the storage device 120, the storage device 320 includes a non-volatile memory and a volatile memory.

[0029] A program 320a1 to be executed by the processing device 300 is stored in the nonvolatile memory of the storage device 320. The volatile memory of the storage device 320 is used by the processing device 300 as a work area when executing the program 320a1. The program 320a1 is obtained, for example, from a server (not shown) connected to the communication network NW via the communication device 310, and then stored in the storage device 320. The program 320a1 may be pre-stored in the storage device 320 at the time of shipping the control device 30A from a factory.

[0030] In addition to the program 320a1, the storage device 320 stores first conversion information 320a2, second conversion information 320a3, an application DB 320a4, and a table 320a5. The first conversion information 320a2 is information for mutual conversion between a position in a captured image and a position in a terminal image. Specific examples of the first conversion information 320a2 include information indicating the position of the center of the terminal image in a coordinate system with the center of the captured image as the origin, and the angle of inclination of the terminal device 40 with respect to the robot 10A.

[0031] The tilt angle will be specifically described with reference to FIG. 5. The Y axis shown in FIG. 5 is a coordinate axis passing through the center P1 of the robot 10A and the center P2 of the imaging range of the imaging device 130. The Z axis in FIG. 5 is a coordinate axis in the vertical direction. The X axis is a coordinate axis perpendicular to the Y and Z axes. The terminal device 40 being tilted with respect to the robot 10A refers to a state in which the long side L of the housing of the terminal device 40 and the Y axis are not parallel to each other in a two-dimensional coordinate system represented by the X and Y axes. Hereinafter, in this embodiment, the angle θ formed between the long side L of the housing of the terminal device 40 and the Y axis is referred to as the tilt angle of the terminal device 40 with respect to the robot 10A.

[0032] Although details will be described later, the tilt angle of the terminal device 40 relative to the robot 10A is detected by the processing device 300 based on an image of the terminal device 40 captured by the imaging device 130. Since the tilt angle of the terminal device 40 is detected by the processing device 300 operating according to the program 320a1, the first conversion information 320a2 is generated by the processing device 300 operating according to the program 320a1. The first conversion information 320a2 generated by the processing device 300 is written to volatile memory.

[0033] The second conversion information 320a3 is information for converting a position in the captured image into a position in a two-dimensional coordinate system represented by the X and Y axes shown in FIG. 5 and having the center position of the robot 10A as the origin. The center position of the robot 10A is the position indicating the center of gravity of the robot 10A when the robot 10A is viewed in a planar view from the positive to the negative Z axis. Hereinafter, the two-dimensional coordinate system represented by the X and Y axes shown in FIG. 5 and having the center position of the robot 10A as the origin will be simply referred to as the "XY coordinate system." Specific examples of the second conversion information 320a3 include information indicating the center position of the imaging range of the imaging device 130 in the XY coordinate system and information indicating the magnification or reduction ratio of the captured image. Note that when the captured image is captured at the same magnification, the second conversion information 320a3 is composed only of information indicating the center position of the imaging range of the imaging device 130 in the XY coordinate system. The second conversion information 320a3 is information determined by the specifications of the robot 10A. The second conversion information 320a3 is stored in advance in the nonvolatile memory of the storage device 320.

[0034] The app DB 320a4 has a plurality of pieces of information indicating the type of app registered in advance, each piece of information being associated with an app icon. In this embodiment, the app DB 320a4 has a pre-registered icon for each app installed on the terminal device 40 and information indicating the type of app. The app DB 320a4 is pre-stored in the non-volatile memory of the storage device 320. The app DB 320a4 is an example of relationship information in the present disclosure. The table 320a5 pre-stores a frame rate suitable for capturing an image of the display area of ​​the terminal device 40 while the app is running, in association with the type of app whose icon is registered in the app DB 320a4. For example, a high frame rate is stored in the table 320a5 for a video viewing app or a game app, and a low frame rate is stored in the table 320a5 for an email app or a SNS app. This is because, while a low frame rate when capturing an image of the terminal device 40 while the video viewing app or the game app is running can interfere with video viewing or game operation on the terminal device 20A, a low frame rate can be used without interfering with remote control for an email app or an SNS app. The table 320a5 is stored in advance in the nonvolatile memory of the storage device 320.

[0035] When control device 30A is powered on, processing device 300 reads program 320a1 from non-volatile memory to volatile memory and starts executing program 320a1. By executing program 320a1, processing device 300 functions as control unit 300a1, acquisition unit 300a2, detection unit 300a3, generation unit 300a4, conversion unit 300a5, identification unit 300a6, and update unit 300a7 shown in Fig. 4. Control unit 300a1, acquisition unit 300a2, detection unit 300a3, generation unit 300a4, conversion unit 300a5, identification unit 300a6, and update unit 300a7 shown in Fig. 4 are software modules realized by operating processing device 300 in accordance with program 320a1. The details of the functions of the control unit 300a1, the acquisition unit 300a2, the detection unit 300a3, the generation unit 300a4, the conversion unit 300a5, the identification unit 300a6, and the update unit 300a7 will be described in an operation example.

[0036] The processing device 300 operates in accordance with the program 320a1 to execute a display operation and a remote control operation. The display operation is an operation of displaying a terminal image on the display device 230 of the terminal device 20A. The display operation is initiated when a captured image is received via the communication device 310. FIG. 6 is a flowchart showing the flow of the display operation. As shown in FIG. 6, the display operation includes a first acquisition process SA110, a detection process SA120, a generation process SA130, and a display control process SA140.

[0037] In the first acquisition process SA110, the processing device 300 functions as the acquisition unit 300a2. That is, the first acquisition process SA110 is a process executed by the acquisition unit 300a2. In the first acquisition process SA110, the acquisition unit 300a2 acquires, from the communication device 310, the captured image transmitted from the robot 10A.

[0038] In the detection process SA120 subsequent to the first acquisition process SA110, the processing device 300 functions as the detection unit 300a3. That is, the detection process SA120 is a process executed by the detection unit 300a3. The detection unit 300a3 detects the inclination of the terminal device 40 with respect to the robot 10A based on the captured image acquired by the acquisition unit 300a2. Specific embodiments of the method for detecting the inclination of the terminal device 40 with respect to the robot 10A include the following first, second, third, and fourth embodiments.

[0039] The first mode is a mode in which the inclination of the terminal device 40 relative to the robot 10A is detected based on the contour of the terminal device 40 or the contour of the display area of ​​the terminal device 40 shown in the captured image. In the first mode, the detection unit 300a3 scans pixel values ​​in each of the X-axis and Y-axis directions of the captured image and extracts points where the pixel values ​​change rapidly as the contour of the terminal device 40. The detection unit 300a3 calculates the difference in pixel values ​​(brightness) between two pixels adjacent in the scanning direction and detects pixels where the calculated difference is equal to or greater than a predetermined value as pixels on the contour. Next, the detection unit 300a3 performs the same process on the inside of the contour of the terminal device 40, and determines the inside of the extracted contour as the display area. In this embodiment, the contour of the terminal device 40 and the contour of the display area of ​​the terminal device 40 form a rectangle. If the captured image is a vertically elongated image in the Y-axis direction, for example, the detection unit 300a3 determines the angle of inclination of the terminal device 40 to be the angle when the long side of the captured image is rotated clockwise so that the long side and the contour line of the terminal device 40 are parallel.

[0040] The second aspect is an aspect in which the tilt of the terminal device 40 is detected based on the shape of an icon shown in the captured image. In the second aspect, the detection unit 300a3 extracts an image of the icon from the image of the display area using the same method as in the first aspect. The image of the icon extracted by the detection unit 300a3 is an example of the second image in the present disclosure. The detection unit 300a3 then determines the angle between the outline of the icon and the outline of the captured image as the tilt angle. If multiple icons are shown in the captured image, the detection unit 300a3 selects an icon with a shape other than a circle and detects the tilt using the method of the first aspect.

[0041] The third aspect is an aspect in which, when a captured image contains multiple icons, tilt is detected based on the arrangement of the multiple icons. In smartphones, multiple icons are often displayed in a matrix along the long and short sides of the display area. Therefore, a line passing through the center point of one of the multiple icons and the center point of the icon closest to the one icon is parallel to the long or short side of the display area. The detection unit 300a3 detects tilt based on the angle formed by the line passing through the center point of one of the multiple icons and the center point of the icon closest to the one icon and the long side of the captured image. An image of multiple icons is also an example of a second image in the present disclosure.

[0042] The fourth aspect is an aspect in which the tilt of the terminal device 40 is identified by using a character string displayed together with the icon and the icon. An icon is often displayed together with a character string indicating the name of an application corresponding to the icon. The angle formed by the direction in which the characters in the character string are arranged and the long side or short side of the outline of the captured image is taken as the tilt angle. An image of the character string displayed together with the icon is also an example of a second image in the present disclosure.

[0043] In this embodiment, the detection unit 300a3 detects the inclination of the terminal device 40 relative to the robot 10A using the detection method of the first aspect. However, the detection unit 300a3 may detect the inclination using the detection method of the second aspect, the third aspect, or the fourth aspect. Furthermore, the detection unit 300a3 may detect the inclination using any two or more of the first aspect, the second aspect, the third aspect, and the fourth aspect, such as attempting to detect the inclination using the detection method of the first aspect and, if the inclination cannot be detected, attempting to detect the inclination using the detection method of the second aspect.

[0044] In generation processing SA130 following detection processing SA120, processing device 300 functions as generation unit 300a4. That is, generation processing SA130 is processing executed by generation unit 300a4. Generation unit 300a4 generates a terminal image based on the captured image acquired by acquisition unit 300a2. More specifically, generation unit 300a4 scans pixel values ​​in each of the Y and X directions of the captured image and determines the location where pixel values ​​change rapidly as the contour line of terminal device 40. Then, generation unit 300a4 performs the same processing on the inside of the contour line of terminal device 40 and determines the inside of the extracted contour line as the display area. Next, generation unit 300a4 corrects the image of the extracted display area by rotating it in the opposite direction by the angle of tilt detected by detection unit 300a3, thereby generating a terminal image.

[0045] For example, assume that the acquisition unit 300a2 has acquired the captured image G1 shown in FIG. 7. In the example shown in FIG. 7, the area hatched with diagonal lines is the display area of ​​the terminal device 40. The black rectangles in FIG. 7 indicate icons displayed in the display area of ​​the terminal device 40. The generation unit 300a4 extracts an image of the display area of ​​the terminal device 40, i.e., the image of the area hatched with diagonal lines in the example shown in FIG. 7, from the captured image shown in FIG. 7, and rotates it in the opposite direction by the angle of inclination detected by the detection unit 300a3 to generate the terminal image G2 shown in FIG. 8. Note that if the generation unit 300a4 can acquire information indicating the size of the display area of ​​the display device 230 from the terminal device 20A through communication via the communication device 310, the generation unit 300a4 may generate the terminal image by enlarging or reducing the image of the display area of ​​the terminal device 40 extracted from the captured image to match the size of the display area of ​​the display device 230. The generation unit 300a4 also generates the first conversion information 320a2. If the terminal image is enlarged or reduced when it is generated, the first conversion information 320a2 also includes information indicating the enlargement or reduction rate of the enlargement, etc. The generation unit 300a4 stores the first conversion information 320a2 in the volatile memory of the storage device 320.

[0046] In display control processing SA140 subsequent to generation processing SA130, processing device 300 functions as control unit 300a1. That is, display control processing SA140 is processing executed by control unit 300a1. Control unit 300a1 transmits a terminal image to terminal device 20A, thereby causing terminal device 20A to display the terminal image. As a result, when a plurality of first icons are displayed in the display area of ​​terminal device 40, a plurality of second icons corresponding one-to-one to the plurality of first icons are displayed in the display area of ​​terminal device 20A. The tilt-corrected terminal image is displayed on display device 230 of terminal device 20A, and touch operations are performed on the tilt-corrected terminal image, improving operability.

[0047] Next, the remote control operation will be described. The remote control operation is an operation for operating the robot 10A in response to a touch operation made on the display area of ​​the display device 230, and causing the robot 10A to execute a touch operation on the terminal device 40. FIG. 9 is a flowchart showing the flow of the remote control operation. This remote control operation is started when second position information transmitted from the terminal device 20A is received via the communication device 310. As shown in FIG. 9, the remote control operation includes a second acquisition process SA210, a conversion process SA220, an identification process SA230, an update process SA240, and an operation control process SA250.

[0048] In the second acquisition process SA210, the processing device 300 functions as an acquisition unit 300a2. That is, the second acquisition process SA210 is a process executed by the acquisition unit 300a2. In the second acquisition process SA210, the acquisition unit 300a2 acquires, from the communication device 310, the second location information transmitted from the terminal device 20A.

[0049] In the conversion process SA220 subsequent to the second acquisition process SA210, the processing device 300 functions as the conversion unit 300a5. That is, the conversion process SA220 is a process executed by the conversion unit 300a5. The conversion unit 300a5 converts the second position information acquired by the acquisition unit 300a2 into first position information using the first conversion information 320a2 and the second conversion information 320a3. The first position information indicates a touch position in the XY coordinate system of a touch operation to be performed on the terminal device 40. More specifically, the conversion unit 300a5 first converts the second position information acquired by the acquisition unit 300a2 into position information indicating a position in the captured image using the first conversion information 320a2. Next, the conversion unit 300a5 converts the position information into first position information indicating a position in the XY coordinate system using the second conversion information 320a3.

[0050] In the identification process SA230 subsequent to the conversion process SA220, the processing device 300 functions as the identification unit 300a6. That is, the identification process SA230 is a process executed by the identification unit 300a6. The identification unit 300a6 identifies a first icon corresponding to a second icon selected by the second touch operation from among the plurality of second icons based on the second position information acquired by the acquisition unit 300a2. The identification unit 300a6 determines the position indicated by the second position information as the center position of the second icon. The identification unit 300a6 converts the center position into a position within the captured image using the first conversion information 320a2. The identification unit 300a6 extracts an image of a predetermined range from the captured image, centered on the converted position, using the first conversion information 320a2. The identification unit 300a6 determines the extracted image as an image of a first icon corresponding to a second icon selected by the second touch operation.

[0051] In the update process SA240 subsequent to the identification process SA230, the processing device 300 functions as the update unit 300a7. That is, the update process SA240 is a process executed by the update unit 300a7. Based on the type of application corresponding to the first icon identified by the identification unit 300a6, the update unit 300a7 updates the settings for imaging by the robot 10A to settings according to the type of application corresponding to the first icon. The update unit 300a7 first identifies the type of application corresponding to the first icon identified by the identification unit 300a6 by referring to the application DB 320a4.

[0052] Next, the updating unit 300a7 refers to the table 320a5 to identify a frame rate according to the type of application corresponding to the first icon identified by the identifying unit 300a6. Then, the updating unit 300a7 causes the robot 10A to update the frame rate of the captured image to the frame rate identified by referring to the table 320a5 by transmitting an update instruction to the robot 10A via the communication device 310.

[0053] In the operation control process SA250 subsequent to the update process SA240, the processing device 300 functions as the control unit 300a1. That is, the operation control process SA250 is a process executed by the control unit 300a1. In the operation control process SA250, the control unit 300a1 transmits an operation instruction including the first position information obtained by the conversion by the conversion unit 300a5 to the robot 10A, thereby causing the robot 10A to perform a touch operation of touching the position indicated by the first position information.

[0054] In the remote control system 1A of this embodiment, the terminal device 20A can perform a touch operation on the terminal image whose tilt with respect to the robot 10A has been corrected, thereby improving operability. Furthermore, in the remote control system 1A of this embodiment, the second position information indicating the touch position with respect to the display area of ​​the terminal device 20A is converted into the first position information using the first conversion information 320a2 and the second conversion information 320a3, thereby improving conversion accuracy and improving operation accuracy of the remote control. Furthermore, in the remote control system 1A of this embodiment, the frame rate when capturing an image of the terminal device 40 is set according to the type of application instructed to be executed by the second touch operation, thereby making it possible to capture an image of the terminal device 40 running an application at a frame rate suitable for that application without unnecessarily increasing communication traffic.

[0055] The first embodiment described above may be modified as follows. (A1) In this embodiment, the robot 10A transmits the image captured by the imaging device 130 directly to the control device 30A. However, the robot 10A may extract a terminal image from the image captured by the imaging device 130 and transmit the extracted terminal image to the control device 30A. (A2) In this embodiment, one processing device 300 functions as control unit 300a1, acquisition unit 300a2, detection unit 300a3, generation unit 300a4, conversion unit 300a5, identification unit 300a6, and update unit 300a7. However, the functions of control unit 300a1, acquisition unit 300a2, detection unit 300a3, generation unit 300a4, conversion unit 300a5, identification unit 300a6, and update unit 300a7 may be shared among a plurality of processing devices that can communicate with each other via communication network NW. (A3) The conversion unit 300a5 may execute a process of converting the second position information into a position in the captured image, and the robot 10A may execute the conversion from the position in the captured image to a position in the XY coordinate system. (A4) If processing device 300 can access application DB 320a4 via communication network NW, application DB 320a4 does not need to be stored in storage device 320. (A5) In this embodiment, the detection unit 300a3 detects the inclination of the terminal device 40 relative to the robot 10A, but the detection unit 300a3 may be omitted. In this case, the generation unit 300a4 generates a terminal image without generating a corrected image. The terminal image is generated, for example, based on the first image. Furthermore, the conversion unit 300a5 converts the second position information into the first position information. This modification can simplify the configuration of the control device 30A. For example, this modification is suitable when the position of the terminal device 40 relative to the robot 10A is predetermined.

[0056] The present disclosure is not limited to the first embodiment and each modified example of the first embodiment described above, and can be realized in various aspects without departing from the spirit thereof. For example, the present disclosure can also be realized in the following aspects. The technical features in the first embodiment corresponding to the technical features in each aspect described below can be replaced or combined as appropriate to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.

[0057] One aspect of the control device of the present disclosure, as understood from the first embodiment and each modified example of the first embodiment described above, is a control device that controls a robot 10A, which performs a first touch operation on a display area of ​​a terminal device 40, where a plurality of first icons corresponding one-to-one to a plurality of applications are displayed, based on a second touch operation on the display area of ​​a terminal device 20A, and includes an acquisition unit 300a2, a control unit 300a1, an identification unit 300a6, and an update unit 300a7. The acquisition unit 300a2 acquires a first image generated by the robot 10A capturing an image of the terminal device 40 and second position information indicating the position in the display area of ​​the terminal device 20A touched by the second touch operation. The control unit 300a1 displays a plurality of second icons corresponding one-to-one to the plurality of first icons on the display area of ​​the terminal device 20A based on the first image, and causes the robot 10A to perform the first touch operation based on the second position information. The identification unit 300a6 identifies a first icon corresponding to the second icon selected by the second touch operation from among the plurality of second icons, based on the second position information. The update unit 300a7 updates the imaging settings of the robot 10A to settings corresponding to the type of application corresponding to the first icon, based on the type of application corresponding to the first icon. In remote control of the terminal device 40 using the robot 10A, the control device 30A of this aspect can set the imaging settings used when the robot 10A captures an image of the display area of ​​the terminal device 40 to settings corresponding to the application executed by remote control.

[0058] In a more preferred embodiment, the control device of the present disclosure may include a detection unit 300a3 and a conversion unit 300a5. The detection unit 300a3 detects the inclination of the terminal device 40 with respect to the robot 10A based on the first image or a second image obtained by extracting a portion of the first image. The conversion unit 300a5 converts the second position information into first position information indicating a position in the display area of ​​the terminal device 40 touched by the first touch operation, based on the inclination detected by the detection unit 300a3. In this embodiment, the identification unit 300a6 identifies the one first icon based on the first position information. The control device 30A of this embodiment can cause the robot 10A to perform a first touch operation on the terminal device 40 based on a second touch operation when the terminal device 40 is inclined with respect to the robot 10A.

[0059] In another preferred embodiment, the update unit 300a7 may identify the type of application corresponding to one of the first icons by referring to relationship information in which the first icons and the application types are associated one-to-one. The control device 30A of this embodiment can identify the type of application corresponding to one of the first icons by referring to relationship information in which the first icons and the application types are associated one-to-one.

[0060] In another preferred embodiment, the image capture settings may include at least one of a resolution and a frame rate. In this embodiment, the control device 30A can set at least one of a resolution and a frame rate when the robot 10A captures an image of the display area of ​​the terminal device 40 in order to remotely control the terminal device 40 using the robot 10A, according to an application to be executed by remote control.

[0061] B. Second embodiment Next, a second embodiment of the present disclosure will be described. FIG. 10 is a diagram illustrating a configuration example of a remote control system 1B according to a second embodiment of the present disclosure. In FIG. 10, the same components as those in FIG. 1 are denoted by the same reference numerals. As is clear from comparing FIG. 10 with FIG. 1, the remote control system 1B differs from the remote control system 1A in that it includes a terminal device 20B instead of the terminal device 20A and does not include the control device 30A. Like the terminal device 20A, the terminal device 20B is an information processing device that accepts user operations for remotely controlling the terminal device 40. The terminal device 20B also serves as the control device 30A. The terminal device 20B is, for example, a smartphone. The following description will focus on the terminal device 20B, which prominently exhibits the features of this embodiment.

[0062] FIG. 11 is a diagram showing an example of the configuration of terminal device 20B. In FIG. 11, the same components as those in FIG. 3 are assigned the same reference numerals. As is clear from comparing FIG. 11 with FIG. 3, the hardware configuration of terminal device 20B is the same as the hardware configuration of terminal device 20A. That is, terminal device 20B includes a processing device 200, a communication device 210, a storage device 220, a display device 230, and an input device 240. Terminal device 20B differs from terminal device 20A in the following two points. The first difference is that program 220b1 is stored in the storage device 220 instead of program 220a1. The second difference is that first conversion information 320a2, second conversion information 320a3, application DB 320a4, and table 320a5 are stored in the storage device 220.

[0063] When the processing device 200 of the terminal device 20B receives from the input device 240 operation data indicating an operation instructing the start of execution of the program 220b1, the processing device 200 reads the program 220b1 from the non-volatile memory to the volatile memory and starts execution of the program 220b1. Like the program 2201, the program 220b1 may be executed by being called from another program. As shown in FIG. 11 , the processing device 200 operating in accordance with the program 220b1 functions as a control unit 300b1, an acquisition unit 300b2, a detection unit 300a3, a generation unit 300a4, a conversion unit 300a5, an identification unit 300b6, and an update unit 300a7. The control unit 300b1, the acquisition unit 300b2, the detection unit 300a3, the generation unit 300a4, the conversion unit 300a5, the identification unit 300b6, and the update unit 300a7 are software modules realized by operating the processing device 200 in accordance with the program 220b1. The functions of the control unit 300b1, the acquisition unit 300b2, and the identification unit 300b6, which are differences from the first embodiment, will be described below in accordance with the operations executed by the processing device 200 in accordance with the program 220b1.

[0064] As in the first embodiment, the operations executed by the processing device 200 of the terminal device 20B according to the program 220b1 are broadly divided into a display operation and a remote control operation. The display operation is an operation of displaying a terminal image on the display device 230. The remote control operation is an operation of operating the robot 10A in response to a touch operation performed on the display area of ​​the display device 230, and causing the robot 10A to perform a touch operation on the terminal device 40.

[0065] Fig. 12 is a flowchart showing the flow of the display operation. The display operation shown in Fig. 12 is started when a captured image is received by communication device 210. In Fig. 12, the same processes as those in Fig. 6 are assigned the same reference numerals. As is clear from comparing Fig. 12 with Fig. 6, the display operation in this embodiment differs from the display operation in the first embodiment in the following two points. The first difference is that display control process SB140 is executed instead of display control process SA140. The second difference is that identification process SB230 and update process SA240 are included.

[0066] In the first acquisition process SA110, the processing device 200 of the terminal device 20B functions as an acquisition unit 300b2. In the first acquisition process SA110, the acquisition unit 300b2 acquires, from the communication device 210, a captured image transmitted from the robot 10A. In the detection process SA120 subsequent to the first acquisition process SA110, the processing device 200 of the terminal device 20B functions as a detection unit 300a3. The detection unit 300a3 detects the inclination of the terminal device 40 relative to the robot 10A based on the captured image acquired by the acquisition unit 300b2. In this embodiment, the detection unit 300a3 detects the inclination of the terminal device 40 relative to the robot 10A using the detection method of the first aspect, as in the first embodiment. In the generation process SA130 subsequent to the detection process SA120, the processing device 200 of the terminal device 20B functions as a generation unit 300a4. The generation unit 300a4 generates a terminal image based on the captured image acquired by the acquisition unit 300b2. The generation unit 300a4 also generates first conversion information 320a2 and stores the generated first conversion information 320a2 in the volatile memory of the storage device 220.

[0067] In the display operation of this embodiment, in the display control process SB140 that follows the generation process SA130, the processing device 200 of the terminal device 20B functions as the control unit 300b1. That is, the display control process SB140 is a process executed by the control unit 300b1. In the display control process SB140, the control unit 300b1 causes the display device 230 to display the terminal image generated in the generation process SA130.

[0068] In the identification process SB230 subsequent to the display control process SB140, the processing device 200 of the terminal device 20B functions as the identification unit 300b6. That is, the identification process SB230 is a process executed by the identification unit 300b6. The identification unit 300b6 identifies one first icon selected by the first touch operation from among a plurality of first icons displayed in the display area of ​​the terminal device 40, based on the captured image acquired in the first acquisition process SA110. More specifically, the identification unit 300b6 identifies the first icon at which the indicator 142 is positioned in the captured image in which the light-emitting unit of the indicator 142 is in an illuminated state, as the one first icon selected by the first touch operation.

[0069] In an update process SA240 subsequent to the identification process SB230, the processing device 200 of the terminal device 20B functions as an update unit 300a7. The update unit 300a7 identifies a frame rate according to the type of application corresponding to the first icon identified by the identification unit 300b6, by referring to the table 320a5. The update unit 300a7 then causes the robot 10A to update the frame rate of the captured image to the frame rate identified by referring to the table 320a5, by transmitting an update instruction to the robot 10A via the communication device 310.

[0070] FIG. 13 is a flowchart showing the flow of the remote control operation in this embodiment. This remote control operation is initiated when a touch operation is detected by the input device 240. In FIG. 13, the same processes as those in FIG. 9 are assigned the same reference numerals. As is clear from comparing FIG. 13 with FIG. 9, the remote control operation in this embodiment differs from the remote control operation in the first embodiment in the following two points. The first difference is that the second acquisition process SB210 is executed instead of the second acquisition process SA210. The second difference is that the identification process SA230 and the update process SA240 are not included. The reason why the identification process SA230 and the update process SA240 are not included in the remote control operation in this embodiment is because the processes corresponding to these processes are included in the display operation.

[0071] In the second acquisition process SB210, the processing device 200 of the terminal device 20B functions as an acquisition unit 300b2. That is, the second acquisition process SB210 is a process executed by the acquisition unit 300b2. In the second acquisition process SB210, the acquisition unit 300b2 acquires, from the input device 240, information indicating a touch operation on the display area of ​​the display device 230, i.e., second position information.

[0072] In the conversion process SA220 subsequent to the second acquisition process SB210, the processing device 200 of the terminal device 20B functions as a conversion unit 300a5. The conversion unit 300a5 converts the second location information acquired by the acquisition unit 300b2 into first location information using the first conversion information 320a2 and the second conversion information 320a3.

[0073] In the operation control process SA250 subsequent to the conversion process SA220, the processing device 200 of the terminal device 20B functions as the control unit 300b1. In the operation control process SA250, the control unit 300b1 transmits an operation instruction including the first position information obtained by the conversion by the conversion unit 300a5 to the robot 10A, thereby causing the robot 10A to perform a touch operation of touching the position indicated by the first position information.

[0074] In this embodiment, the terminal device 20B can also perform a touch operation on a terminal image whose tilt with respect to the robot 10A has been corrected, thereby improving operability. Furthermore, in this embodiment, the second position information indicating the touch position with respect to the display area of ​​the terminal device 20B is converted into the first position information using the first conversion information 320a2 and the second conversion information 320a3, thereby improving the conversion accuracy and the operation accuracy of remote control. Furthermore, in the remote control system 1B of this embodiment, the frame rate at which the robot 10A captures an image of the terminal device 40 is set according to the type of application instructed to be executed by the first touch operation. Therefore, it is possible to capture an image of the terminal device 40 running an application at a frame rate suitable for the application without unnecessarily increasing communication traffic. In addition, according to this embodiment, the cost of building the remote control system 1B can be reduced compared to the cost of building the remote control system 1A by the amount corresponding to the absence of the control device 30A.

[0075] The second embodiment described above may be modified as follows. (B1) The application to be executed by the terminal device 40 may be identified based on the second location information acquired by the acquisition unit 300b2. The aspect of identifying the application to be executed by the terminal device 40 based on the second location information is realized by replacing the identification unit 300b6 with the identification unit 300a6. (B2) The identification unit 300b6 detects that the light-emitting unit of the indicator 142 has entered an emitting state in the captured image, and identifies the first icon located at the position of the detected light-emitting unit as the first icon selected by the first touch operation; however, this is not limited to this. The movement mechanism moves the indicator 142 downward in the first touch operation. During this movement, when the indicator 142 comes into contact with the terminal device 40, the movement mechanism detects a reaction force and outputs a detection signal indicating the detection. When the detection signal is valid, the identification unit 300b6 may detect the position of the indicator 142 based on the captured image, and identify the first icon located at the detected position as the first icon selected by the first touch operation. (B3) In this embodiment, the detection unit 300a3 detects the inclination of the terminal device 40 relative to the robot 10A, but the detection unit 300a3 may be omitted. In this case, the generation unit 300a4 generates a terminal image without generating a corrected image. The terminal image is generated, for example, based on the first image. Furthermore, the conversion unit 300a5 converts the second position information into the first position information. This modification simplifies the configuration of the terminal device 20B, in other words, the control device that causes the robot 10A to perform the first touch operation in response to the second touch operation. For example, this modification is suitable when the position of the terminal device 40 relative to the robot 10A is predetermined.

[0076] One aspect of the control device of the present disclosure, understood from the second embodiment and each modified example of the second embodiment described above, is a control device that controls a robot 10A, which performs a first touch operation on a display area of ​​a terminal device 40, where multiple first icons corresponding one-to-one to multiple applications are displayed, based on a second touch operation on a display area of ​​a terminal device 20B, and includes an acquisition unit 300b2, a control unit 300b1, an identification unit 300b6, and an update unit 300a7. The acquisition unit 300b2 acquires a first image generated by the robot 10A capturing an image of the terminal device 40 and second position information indicating the position in the display area of ​​the terminal device 20B touched by the second touch operation. Based on the first image, the control unit 300b1 displays multiple second icons corresponding one-to-one to the multiple first icons in the display area of ​​the terminal device 20B, and causes the robot 10A to perform the first touch operation based on the second position information. Based on the first image, the identification unit 300b6 identifies one of the multiple first icons selected by the first touch operation. The update unit 300a7 updates the imaging settings of the robot 10A to settings according to the type of application corresponding to the one first icon, based on the type of application corresponding to the one first icon. In remote control of the terminal device 40 using the robot 10A, the control device of this aspect can set the imaging settings when the robot 10A captures an image of the display area of ​​the terminal device 40 to settings according to the application executed by remote control.

[0077] A more preferred embodiment of the control device of the present disclosure may include a detection unit 300a3 and a conversion unit 300a5. The detection unit 300a3 detects the tilt of the terminal device 40 with respect to the robot 10A based on the first image or a second image obtained by extracting a portion of the first image. The conversion unit 300a5 converts the second position information into first position information indicating a position in the display area of ​​the terminal device 40 that is touched by the first touch operation, based on the tilt detected by the detection unit 300a3. In this embodiment, the control unit 300b1 causes the robot 10A to perform the first touch operation based on the first position information. The control device of this embodiment can cause the robot 10A to perform the first touch operation with respect to the terminal device 40 based on the second touch operation, when the terminal device 40 is tilted with respect to the robot 10A.

[0078] 3. Third embodiment Next, a third embodiment of the present disclosure will be described. FIG. 14 is a diagram illustrating a configuration example of a remote operation system 1C according to a third embodiment of the present disclosure. In FIG. 14, the same components as those in FIG. 1 are denoted by the same reference numerals. As is clear from comparing FIG. 14 with FIG. 1, the remote operation system 1C differs from the remote operation system 1A in that it includes a robot 10C instead of the robot 10A and does not include the control device 30A. The robot 10C is the same as the control device 30A in that it performs touch operations on the terminal device 40 in response to user operations on the terminal device 20A. The robot 10C differs from the robot 10A in that it also fulfills the role of the control device 30A. The robot 10C also differs from the robot 10A in that it autonomously updates the settings of the imaging device 130. The following description focuses on the robot 10C, which prominently exhibits the features of this embodiment.

[0079] FIG. 15 is a diagram showing an example of the configuration of the robot 10C. In FIG. 15, the same components as those in FIG. 2 are denoted by the same reference numerals. As is clear from comparing FIG. 15 with FIG. 2, the hardware configuration of the robot 10C is the same as that of the robot 10A. That is, the robot 10C includes a processing device 100, a communication device 110, a storage device 120, an imaging device 130, and an operation device 140. The robot 10C differs from the robot 10A in the following two points. The first difference is that a program 120c1 is stored in the storage device 120 instead of the program 120a1. The second difference is that first conversion information 320a2, second conversion information 320a3, an application DB 320a4, and a table 320a5 are stored in the storage device 120.

[0080] When the robot 10C is powered on, the processing device 100 of the robot 10C reads the program 120c1 from the non-volatile memory to the volatile memory and starts executing the program 120c1. The processing device 100 operating in accordance with the program 120c1 functions as a control unit 300c1, an acquisition unit 300c2, a detection unit 300a3, a generation unit 300a4, a conversion unit 300a5, an identification unit 300b6, and an update unit 300c7, as shown in FIG. 15. The control unit 300c1, the acquisition unit 300c2, the detection unit 300a3, the generation unit 300a4, the conversion unit 300a5, the identification unit 300b6, and the update unit 300c7 are software modules realized by operating the processing device 100 in accordance with the program 120c1. The functions of the control unit 300c1, the acquisition unit 300c2, and the update unit 300c7, which are differences from the first and second embodiments, will be described below in accordance with the operations executed by the processing device 100 in accordance with the program 120c1.

[0081] The operations that the processing device 100 of the robot 10C executes in accordance with the program 120c1 are roughly divided into display operations and remote control operations, as in the first and second embodiments.

[0082] Fig. 16 is a flowchart showing the flow of the display operation in this embodiment. This display operation is initiated when the imaging device 130 captures an image of the terminal device 40. In Fig. 16, the same processes as in Fig. 6 are assigned the same reference numerals. As is clear from comparing Fig. 16 with Fig. 6, the display operation in this embodiment differs from the display operation in the first embodiment in the following two points. The first difference is that the first acquisition process SC110 is executed instead of the first acquisition process SA110. The second difference is that the identification process SB230 and the update process SC240 are included.

[0083] In the first acquisition process SC110, the processing device 100 of the robot 10C functions as the acquisition unit 300c2. That is, the first acquisition process SC110 is a process executed by the acquisition unit 300c2. In the first acquisition process SC110, the acquisition unit 300c2 acquires a captured image from the imaging device 130.

[0084] In a detection process SA120 subsequent to the first acquisition process SC110, the processing device 100 of the robot 10C functions as a detection unit 300a3. The detection unit 300a3 detects the inclination of the terminal device 40 relative to the robot 10C based on the captured image acquired by the acquisition unit 300c2. In this embodiment, the detection unit 300a3 detects the inclination of the terminal device 40 relative to the robot 10C using the first detection method, as in the first embodiment. In a generation process SA130 subsequent to the detection process SA120, the processing device 100 of the robot 10C functions as a generation unit 300a4. The generation unit 300a4 generates a terminal image in which the inclination detected by the detection unit 300a3 has been corrected based on the captured image acquired by the acquisition unit 300c2. The generation unit 300a4 also generates first conversion information 320a2 and stores the generated first conversion information 320a2 in the volatile memory of the storage device 120.

[0085] In the display control process SA140 subsequent to the generation process SA130, the processing device 100 of the robot 10C functions as the control unit 300c1. That is, the display control process SA140 in this embodiment is a process executed by the control unit 300c1. In the display control process SA140, the control unit 300c1 transmits the terminal image generated in the generation process SA130 to the terminal device 20A via the communication device 110, thereby causing the terminal image to be displayed on the terminal device 20A. When multiple first icons are displayed in the display area of ​​the terminal device 40, multiple second icons corresponding one-to-one to the multiple first icons are displayed in the display area of ​​the terminal device 20A. In the terminal device 20A, the tilt-corrected terminal image is displayed on the display device 230. In this embodiment, a touch operation is performed on the tilt-corrected terminal image, improving operability.

[0086] In the identification process SB230 subsequent to the display control process SA140 in the display operation of this embodiment, the processing device 100 of the robot 10C functions as an identification unit 300b6. The identification unit 300b6 identifies one first icon selected by the first touch operation from among the multiple first icons displayed in the display area of ​​the terminal device 40, based on the captured image acquired in the first acquisition process SA110.

[0087] In the update process SC240 subsequent to the identification process SB230 in the display operation of this embodiment, the processing device 100 of the robot 10C functions as the update unit 300c7. That is, the update process SC240 is a process executed by the update unit 300c7. In the update process SC240, the update unit 300c7 identifies a frame rate according to the type of application corresponding to the first icon identified by the identification unit 300b6. This identification method is the same as the identification method in the update process SA240 in the first embodiment. The update unit 300c7 updates the frame rate of the captured image to the frame rate identified by the identification unit 300b6.

[0088] FIG. 17 is a flowchart showing the flow of the remote control operation in this embodiment. This remote control operation is executed when second position information is received via communication device 110. In FIG. 17, the same processes as those in FIG. 9 are assigned the same reference numerals. As is clear from comparing FIG. 17 with FIG. 9, the remote control operation in this embodiment differs from the remote control operation in the first embodiment in the following two points. The first difference is that an operation control process SC250 is executed instead of an operation control process SA250. The second difference is that the identification process SA230 and the update process SA240 are not included. The reason why the identification process SA230 and the update process SA240 are not included in the remote control operation in this embodiment is because the processes corresponding to these processes are included in the display operation.

[0089] In the second acquisition process SA210 in the remote control operation of this embodiment, the processing device 100 of the robot 10C functions as an acquisition unit 300c2. That is, the second acquisition process SA210 in this embodiment is a process executed by the acquisition unit 300c2. In the second acquisition process SA210, the acquisition unit 300c2 acquires the second position information transmitted from the terminal device 20A from the communication device 310. In the conversion process SA220 subsequent to the second acquisition process SA210, the processing device 100 of the robot 10C functions as a conversion unit 300a5. The conversion unit 300a5 converts the second position information acquired by the acquisition unit 300c2 into first position information using the first conversion information 320a2 and the second conversion information 320a3.

[0090] In the remote-operation operation of this embodiment, in the operation control process SC250 executed following the conversion process SA220, the processing device 100 of the robot 10C functions as the control unit 300c1. That is, the operation control process SC250 is a process executed by the control unit 300c1. In the operation control process SC250, the control unit 300c1 controls the drive of the operation device 140 in accordance with the first position information converted in the conversion process SA220, and touches the position indicated by the first position information.

[0091] According to the present embodiment, the operability and accuracy of remote control are improved, as in the first embodiment. Furthermore, in the present embodiment, the robot 10C can be made to autonomously update the frame rate of the captured image in accordance with the type of application to be executed by the first touch operation. In addition, according to this embodiment, the cost of building the remote control system 1C can be reduced compared to the cost of building the remote control system 1A by the amount corresponding to the absence of the control device 30A.

[0092] The third embodiment described above may be modified as follows. (C1) The application to be executed by the terminal device 40 may be identified based on the second location information acquired by the acquisition unit 300b2. The aspect of identifying the application to be executed by the terminal device 40 based on the second location information is realized by replacing the identification unit 300b6 with the identification unit 300a6. (C2) When setting information indicating the frame rate of the captured image is transmitted from the terminal device 20A, the acquisition unit 300c2 may acquire the setting information by receiving the setting information transmitted from the terminal device 20A via the communication device 110. When the acquisition unit 300c2 acquires the setting information transmitted from the terminal device 20A, the update unit 300c7 may update the frame rate of the captured image based on the setting information acquired by the acquisition unit 300c2. (C3) In this embodiment, the detection unit 300a3 detects the inclination of the terminal device 40 relative to the robot 10C, but the detection unit 300a3 may be omitted. In this case, the generation unit 300a4 generates a terminal image without generating a corrected image. The terminal image is generated, for example, based on the first image. Furthermore, the conversion unit 300a5 converts the second position information into the first position information. This modification can simplify the configuration of the robot 10C. For example, this modification is suitable when the position of the terminal device 40 relative to the robot 10C is predetermined.

[0093] One aspect of the robot of the present disclosure that can be understood from the third embodiment and each modified example of the third embodiment described above is a robot 10C that executes a first touch operation on a display area of ​​a terminal device 40 based on a second touch operation on a display area of ​​a terminal device 20A, and includes an imaging device 130, an operation device 140, and a processing device 100. The operation device 140 is located between the terminal device 40 and the imaging device 130 and performs the first touch operation. The processing device 100 includes an acquisition unit 300c2, a control unit 300c1, and an update unit 300c7. The acquisition unit 300c2 acquires instruction information related to the first touch operation and setting information related to imaging settings of the imaging device 130. The control unit 300c1 causes the operation device 140 to execute the first touch operation based on the instruction information. The update unit 300c7 updates the setting information of the imaging device 130 based on the setting information. The robot 10C of this aspect updates the setting information of the imaging device 130 based on the setting information acquired by the acquisition unit 300c2. If the setting information acquired by the acquisition unit 300c2 is based on an application to be executed by the terminal device 40 by remote control, the imaging settings when the robot 10C captures an image of the display area of ​​the terminal device 40 are updated to settings corresponding to the application to be executed by the terminal device 40.

[0094] Another aspect of the robot of the present disclosure is a robot 10C that performs a first touch operation on a display area of ​​a terminal device 40, where a plurality of first icons corresponding one-to-one to a plurality of applications are displayed, based on a second touch operation on a display area of ​​a terminal device 20A. The robot 10C includes an imaging device 130, an operating device 140, and a processing device 100. The operating device 140 is located between the terminal device 40 and the imaging device 130 and performs the first touch operation. The processing device 100 includes an acquiring unit 300c2, a control unit 300c1, an identifying unit 300b6, and an updating unit 300c7. The acquiring unit 300c2 acquires instruction information related to the first touch operation. The control unit 300c1 causes the operating device 140 to perform the first touch operation based on the instruction information. The identifying unit 300b6 identifies one of the plurality of first icons selected by the first touch operation based on a first image showing the display area of ​​the terminal device 40 captured by the imaging device 130. The update unit 300a7 updates the settings for imaging by the imaging device 130 to settings according to the type of application corresponding to the one first icon, based on the type of application corresponding to the one first icon. The robot 10C of this embodiment updates the settings for imaging when capturing an image of the display area of ​​the terminal device 40 for remote control of the terminal device 40 to settings according to the application to be executed by remote control.

[0095] In a more preferred embodiment of the robot 10C, the update unit 300c7 identifies the type of application corresponding to one of the first icons by referring to relationship information that associates the first icons with the application types in a one-to-one correspondence. The robot 10C of this embodiment can identify the type of application corresponding to one of the first icons based on the relationship information that associates the first icons with the application types in a one-to-one correspondence.

[0096] In a more preferred embodiment of the robot 10C, the operating device 140 may include an indicator 142 for inputting to the terminal device 40, and an arm 144 for moving the indicator 142 within the imaging range of the imaging device 130. When the operating device 140 includes the indicator 142 and the arm 144 for moving the indicator 142, the arm 144 is preferably made of a transparent material. The robot 10C of this embodiment can prevent the arm 144 from interfering with the imaging device 130 capturing an image of the terminal device 40.

[0097] D. Other Variations Each embodiment may be further modified as follows. (D1) In the above embodiments, the terminal device 40 is a smartphone, but it may be a tablet terminal. Similarly, the terminal device 20A in the first and third embodiments may be a tablet terminal. The terminal device 20B in the second embodiment may also be a tablet terminal. Some or all of the control unit 300a1, acquisition unit 300a2, detection unit 300a3, generation unit 300a4, conversion unit 300a5, identification unit 300a6, and update unit 300a7 in the first embodiment may be hardware. Examples of this hardware include a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). Even if some or all of the control unit 300a1, acquisition unit 300a2, detection unit 300a3, generation unit 300a4, conversion unit 300a5, identification unit 300a6, and update unit 300a7 are hardware, the same effects as those of the first embodiment can be achieved. Similarly, the control units 100a1, 200a1, 300b1, and 300c1 may also be hardware. Furthermore, the acquisition unit 300b2, acquisition unit 300c2, identification unit 300b6, and update unit 300c7 may also be hardware. That is, each function disclosed in the above embodiment can be realized by any combination of hardware and software. Also, each function can be realized by a single device or by two or more devices configured separately from each other.

[0098] (D2) The detection unit 300a3 may detect the tilt of the terminal device 40 by matching the icon of each app stored in the app DB 320a4 with the image of the icon extracted from the captured image of the terminal device 40. Specifically, the detection unit 300a3 performs pattern matching with the image of each icon stored in the app DB 320a4 while rotating the image of the icon extracted from the captured image by a predetermined angle, and determines the angle when the image matches any of the images of the multiple icons stored in the app DB 320a4 as the tilt angle of the terminal device 40.

[0099] (D3) In the first embodiment, the program 320a1 is stored in the storage device 320. However, the program 320a1 may be manufactured or distributed as a standalone program. Specific methods for distributing the program 320a1 include writing the program 320a1 to a computer-readable recording medium such as a flash ROM (Read Only Memory) and distributing it, or distributing it by downloading it via a telecommunications line such as the Internet. By installing the program 320a1 in a general information processing device such as a personal computer and operating a computer such as the CPU of the information processing device in accordance with the program 320a1, the information processing device can function as the control device of the present disclosure. The program 220b1 and the program 120c1 may also be manufactured or distributed as standalone programs.

[0100] (D4) In the above embodiments, storage device 120 is a recording medium readable by processing device 100. ROM and RAM are exemplified as examples, but storage device 120 may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The same applies to storage device 220 and storage device 320. Furthermore, the program may be transmitted from a network via a telecommunications line. Furthermore, the program may be transmitted from a communications network via a telecommunications line. The base station may include an input device such as an input key and an output device such as a display.

[0101] (D5) The above embodiments may be applied to systems using LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), or other suitable systems, and / or next generation systems enhanced based on these.

[0102] (D6) In each of the above embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof. Note that terms described in this specification and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0103] (D7) In each of the above embodiments, input and output information may be stored in a specific location (for example, memory) or managed in a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0104] (D8) In each of the above embodiments, the judgment may be made based on a value represented by one bit (0 or 1), a boolean value (true or false), or a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0105] (D9) The programs exemplified in the above embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, or functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names. Furthermore, software, instructions, etc. may be transmitted and received via a transmission medium. For example, when software is transmitted from a website, server, or other remote source using wired technologies such as coaxial cable, optical fiber cable, twisted pair, and digital subscriber line (DSL), and / or wireless technologies such as infrared, radio, and microwave, these wired and / or wireless technologies are included within the definition of a transmission medium.

[0106] (D10) In each of the above embodiments, the terminal device 20A, 20B, or 40 may be a mobile station, which may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

[0107] (D11) In each of the above embodiments, the term "connected," or any variation thereof, means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" to each other. The connection between elements may be physical, logical, or a combination thereof. As used herein, two elements may be considered to be "connected" to each other by using one or more wires, cables, and / or printed electrical connections, as well as by using electromagnetic energy, such as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0108] (D12) In each of the above embodiments, unless otherwise specified, the phrase "based on" does not mean "based only on." In other words, the phrase "based on" means both "based only on" and "based at least on."

[0109] (D13) To the extent that the words "including," "comprising," and variations thereof are used herein or in the claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used herein or in the claims is not intended to be an exclusive or.

[0110] (D14) Throughout this application, where articles have been added by translation, such as a, an and the in English, these articles include the plural unless the context clearly indicates otherwise.

[0111] (D15) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is for illustrative purposes only and does not have any limiting meaning on the present invention. Furthermore, multiple embodiments selected from the embodiments exemplified in this specification may be combined. [Explanation of symbols]

[0112] 1A, 1B, 1C... Remote operation system, 10A, 10C... Robot, 20A, 20B, 40... Terminal device, 30A... Control device, 100, 200, 300... Processing device, 110, 210, 310... Communication device, 120, 220, 320... Storage device, 130... Imaging device, 140... Operation device, 142... Pointer, 144... Arm, 230... Display device, 240... Input device, 100a 1,200a1, 300a1, 300b1, 300c1...control unit, 300a2, 300b2, 300c2...acquisition unit, 300a3...detection unit, 300a4...generation unit, 300a5...conversion unit, 300a6, 300b6...identification unit, 300a7, 300c7...update unit, 320a2...first conversion information, 320a3...second conversion information, 320a4...application DB, 320a5...table.

Claims

1. A control device that controls a robot that performs a first touch operation on a display area of ​​a first terminal device, the first touch operation being performed on a display area of ​​a second terminal device, the display area including a plurality of first icons that correspond one-to-one to a plurality of applications, based on a second touch operation being performed on a display area of ​​a second terminal device, an acquisition unit that acquires a first image generated by the robot capturing an image of a first terminal device and second position information indicating a position in a display area of ​​the second terminal device touched by the second touch operation; a control unit that displays a plurality of second icons in a one-to-one correspondence with the plurality of first icons in a display area of ​​the second terminal device based on the first image, and causes the robot to execute the first touch operation based on the second position information; an identification unit that identifies one first icon corresponding to one second icon selected by the second touch operation among the plurality of second icons based on the second position information; an update unit that updates a setting for imaging by the robot to a setting according to the type of the application corresponding to the one of the first icons, based on the type of the application corresponding to the one of the first icons; A control device comprising:

2. a detection unit that detects an inclination of the first terminal device with respect to the robot based on the first image or a second image obtained by extracting a portion of the first image; a conversion unit that converts the second position information into first position information indicating a position in a display area of ​​the first terminal device touched by the first touch operation based on the tilt, The identification unit identifies the one of the first icons based on the first position information. The control device according to claim 1 .

3. A control device that controls a robot that performs a first touch operation on a display area of ​​a first terminal device, the first touch operation being performed on a display area of ​​a second terminal device, the display area including a plurality of first icons that correspond one-to-one to a plurality of applications, based on a second touch operation being performed on a display area of ​​a second terminal device, an acquisition unit that acquires a first image generated by the robot capturing an image of a first terminal device and second position information indicating a position in a display area of ​​the second terminal device touched by the second touch operation; a control unit that displays a plurality of second icons in a one-to-one correspondence with the plurality of first icons in a display area of ​​the second terminal device based on the first image, and causes the robot to execute the first touch operation based on the second position information; an identification unit that identifies one of the plurality of first icons selected by the first touch operation based on the first image; an update unit that updates a setting for imaging by the robot to a setting according to the type of the application corresponding to the one of the first icons, based on the type of the application corresponding to the one of the first icons; A control device comprising:

4. a detection unit that detects an inclination of the first terminal device with respect to the robot based on the first image or a second image obtained by extracting a portion of the first image; a conversion unit that converts the second position information into first position information indicating a position in a display area of ​​the first terminal device touched by the first touch operation based on the tilt, The control unit causes the robot to perform the first touch operation based on the first position information. The control device according to claim 3.

5. 5. The control device according to claim 1, wherein the update unit is configured to identify the type of application corresponding to one of the first icons by referring to relationship information that associates the first icons with the types of the applications in a one-to-one relationship.

6. The control device according to claim 1 , wherein the imaging settings include at least one of a resolution and a frame rate.

7. A robot that performs a first touch operation on a display area of ​​a first terminal device on which a plurality of first icons corresponding one-to-one to a plurality of applications are displayed, based on a second touch operation on a display area of ​​a second terminal device, An imaging device; an operation device that is located between the first terminal device and the imaging device and performs the first touch operation; A processing device, The processing device includes: an acquisition unit that acquires instruction information related to the first touch operation; a control unit that causes the operation device to execute the first touch operation based on the instruction information; an identification unit that identifies one of the plurality of first icons selected by the first touch operation based on a first image showing a display area of ​​the first terminal device captured by the imaging device; an update unit that updates a setting for imaging by the imaging device to a setting according to the type of the application corresponding to the one of the first icons, based on the type of the application corresponding to the one of the first icons; A robot equipped with the above.

8. The robot described in Claim 7, wherein the update unit identifies the type of application corresponding to one of the first icons by referring to relationship information that associates the multiple first icons with the multiple application types in a one-to-one manner.

9. The operating device comprises: An indicator for performing an input to the first terminal device; an arm that moves the indicator within an imaging range of the imaging device, The arm is made of a transparent material. The robot according to claim 7 or claim 8.

Images