Information processing device and computer readable medium storing program

Abstract

The present invention is an information processing device and a computer-readable medium storing a program, which can operate an object existing in a real space without using an operation performed by contact of a user. The information processing device used by contact with a user includes: a detection unit that detects the user's action when operating an object existing in a real space as an operation object from an opposite position, that is, the user's action performed in the real space without contact with the object; and a control unit that instructs the object to perform an operation corresponding to the detected user's action.

Description

Technical Field

[0001] The present invention relates to an information processing device and a computer-readable medium storing a program. Background Art

[0002] It has been proposed to recognize a device included in an image captured by a camera built into a portable device operated by a user, display a virtual user interface corresponding to the recognized device in an augmented reality space, and operate the device through the virtual user interface.

[0003] [Prior art literature]

[0004] [Patent Document]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2013-172432 Summary of the invention

[0006] [Problems to be solved by the invention]

[0007] The operation of the device is premised on instructions given by contact of the user.

[0008] The present invention can operate objects existing in a real space instead of using operations performed by contact of a user.

[0009] [Technical means to solve the problem]

[0010] The invention described in technical solution 1 is an information processing device, comprising: a detection component, which detects the user's actions when operating an object that exists in the real space and serves as an object of operation from an opposite position, that is, the user's actions performed in the real space without contacting the object; and a control component, which instructs the object to execute an operation corresponding to the detected user's action; and the information processing device is used in contact with the user.

[0011] The invention described in claim 2 is the information processing device described in claim 1, wherein the detection component detects the user's movement based on an image obtained by simultaneously photographing the object and a part of the user's body.

[0012] The invention described in claim 3 is the information processing device described in claim 2, wherein the photographer of the image is the user.

[0013] The invention described in claim 4 is the information processing device described in claim 3, wherein a part of the user's body is displayed by being superimposed on the object.

[0014] The invention described in Technical Solution 5 is the information processing device described in Technical Solution 2, wherein the control component associates the detected user's movement and the object detected to overlap with a part of the user's body on the image.

[0015] The invention described in claim 6 is the information processing device described in claim 5, wherein the control unit uses a predetermined part of the body as a detection object.

[0016] The invention described in claim 7 is the information processing device described in claim 6, wherein the predetermined body part is a finger or a hand.

[0017] The invention described in claim 8 is the information processing device described in claim 6, wherein the predetermined body part is a foot.

[0018] The invention described in claim 9 is the information processing device described in claim 1, wherein the detection component detects the user's movement when the object and a part of the user's body are simultaneously photographed.

[0019] The invention described in claim 10 is the information processing device described in claim 1, wherein the operation corresponding to the user's action is prepared for each individual part of the object.

[0020] The invention described in claim 11 is the information processing device described in claim 1, wherein the control unit instructs the object to perform an operation as long as the user whose action is detected is a user permitted to operate the object.

[0021] The invention described in claim 12 is the information processing device described in claim 1, wherein when an operation corresponding to the detected action of the user cannot be performed in the object, a report is made to the user.

[0022] The invention described in technical solution 13 is the information processing device described in technical solution 12, wherein a method for reporting the correct operation of the object.

[0023] The invention described in claim 14 is the information processing device described in claim 1, wherein the control unit determines the content of the operation corresponding to the detected user's action based on the object.

[0024] The invention described in claim 15 is the information processing device described in claim 1, wherein the control unit obtains the function of the object that can be operated through communication with the object.

[0025] The invention described in claim 16 is the information processing device described in claim 1, wherein the control component transparently displays the user's body so that the object is not blocked by the user's body.

[0026] The invention described in claim 17 is the information processing device described in claim 1, wherein the control component selectively displays a part of the user's body so that the object is not blocked by the user's body.

[0027] The invention described in claim 18 is the information processing device described in claim 1, wherein the control unit displays a representation suggesting a part of the user's body superimposed on the object.

[0028] The invention described in claim 19 is the information processing device described in claim 1, wherein the control unit reports to the user that the user's action has been accepted as an operation.

[0029] The invention described in claim 20 is the information processing device described in claim 19, wherein the control unit displays the part of the body accepted as the operation in a form different from that before acceptance.

[0030] The invention described in claim 21 is the information processing device described in claim 1, wherein the control unit displays the equipment for operating the object by superimposing it on the user's body.

[0031] The invention described in technical solution 22 is the information processing device described in technical solution 1, wherein the control component superimposes the equipment corresponding to the environment in which the object exists on the user's body for display.

[0032] The invention described in Technical Solution 23 is the information processing device described in Technical Solution 1, wherein the detection component detects the movement of a part of the user's body based on an image obtained by simultaneously photographing a screen showing the image of the object and a part of the user's body.

[0033] The invention described in Technical Solution 24 is the information processing device described in Technical Solution 1, wherein the control component determines the content of the operation corresponding to the user's action based on an image synthesized by an image obtained by photographing the object and an image obtained by photographing a part of the user's body.

[0034] The invention described in Technical Solution 25 is the information processing device described in Technical Solution 1, wherein when an operation corresponding to a user's action requires physical operation of the object, the control component instructs another device having a movable part to execute the operation.

[0035] The invention described in technical solution 26 is an information processing device, which includes a control component, which allows the execution of the indicated operation when receiving a communication indicating the execution of an operation corresponding to the user's action performed in a real space without contact with the device from a terminal used in contact with the user.

[0036] The invention described in claim 27 is the information processing device described in claim 26, wherein the control unit allows the execution of the operation as long as the instruction comes from the terminal.

[0037] The invention described in technical solution 28 is a program that enables a computer used in contact with a user to function as a detection component and a control component, wherein the detection component detects from the user's location the user's actions when operating an object that serves as an object of operation in the real space from an opposite position, that is, the user's actions performed in the real space without contacting the object, and the control component instructs the object to execute an operation corresponding to the detected user's actions.

[0038] [Effects of the Invention]

[0039] According to the invention described in claim 1, it is possible to operate an object existing in a real space without using an operation performed by contact of a user.

[0040] According to the invention described in claim 2, the object can be operated by the same action as that in the case of utilizing contact.

[0041] According to the invention described in claim 3, the object can be operated by the same action as that in the case of utilizing contact.

[0042] According to the invention described in claim 4, the user can confirm the positional relationship between his / her own action and the object to be operated on the screen.

[0043] According to the invention described in claim 5, the object can be operated by the same action as that in the case of utilizing contact.

[0044] According to the invention described in claim 6, erroneous operations can be reduced.

[0045] According to the invention described in claim 7, erroneous operations can be reduced.

[0046] According to the invention described in technical solution 8, erroneous operations can be reduced.

[0047] According to the invention described in technical solution 9, erroneous operations can be reduced.

[0048] According to the invention described in claim 10, the operation efficiency can be improved.

[0049] According to the invention described in claim 11, the operator can be limited.

[0050] According to the invention described in claim 12, incorrect operation can be notified.

[0051] According to the invention described in claim 13, correct operation can be prompted.

[0052] According to the invention described in technical solution 14, an operation corresponding to an object can be indicated.

[0053] According to the invention described in technical solution 15, operations corresponding to the object can be indicated.

[0054] According to the invention described in claim 16, operations can be performed while the object is confirmed on the screen.

[0055] According to the invention described in claim 17, operations can be performed while the object is confirmed on the screen.

[0056] According to the invention described in technical solution 18, erroneous operations can be reduced.

[0057] According to the invention described in claim 19, the determination of the operation can be confirmed.

[0058] According to the invention described in claim 20, the determination of the operation can be confirmed.

[0059] According to the invention recorded in technical solution 21, the sense of reality can be improved.

[0060] According to the invention described in technical solution 22, the sense of reality can be improved.

[0061] According to the invention described in claim 23, even if there is no object to be operated in front of the user, the object can be operated by the same action as in the case of using contact.

[0062] According to the invention described in claim 24, even if there is no object to be operated in front of the user, the object can be operated by the same action as in the case of using contact.

[0063] According to the invention described in technical solution 25, it is possible to instruct to perform operations that also require physical contact.

[0064] According to the invention described in claim 26, it is possible to operate an object existing in a real space without utilizing an operation performed by contact of a user.

[0065] According to the invention recorded in technical solution 27, the safety of operation can be improved.

[0066] According to the invention described in claim 28, it is possible to operate an object existing in a real space without utilizing an operation performed by contact of a user. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a diagram for explaining an example of a system according to the first embodiment.

[0068] Figure 2 This is a diagram for explaining an example of the hardware configuration of the image forming apparatus.

[0069] Figure 3 This is a diagram for explaining an example of the hardware configuration of a smartphone.

[0070] Figure 4 This is a diagram for explaining an example of a functional configuration realized by the execution of a program by a processing circuit unit.

[0071] Figure 5 This is a flowchart illustrating an example of processing operations executed in the processing circuit unit.

[0072] Figure 6 This is a diagram for explaining how a user as a photographer takes an image so that his or her index finger overlaps with the image forming device.

[0073] Figure 7 This is a diagram for explaining how an image is captured of an action such as pressing the index finger in the air without contact with the image forming device.

[0074] Figure 8 This is a diagram for explaining another method of reporting the acceptance of an operation to the user.

[0075] Fig. 9 This is a diagram for explaining another method of reporting the acceptance of an operation to the user.

[0076] Fig.10 This is a diagram for explaining another method of reporting the acceptance of an operation to the user.

[0077] Fig.11 A diagram for explaining a method of preventing the image of the index finger from being displayed on a liquid crystal display.

[0078] Fig.12 This is a diagram for explaining another method of preventing the image of the index finger from being displayed on a liquid crystal display.

[0079] Fig.13 This is a diagram for explaining another method of preventing the image of the index finger from being displayed on a liquid crystal display.

[0080] Fig.14 This is a diagram for explaining a case where the object of operation is a calculator having a communication function.

[0081] Fig.15 This is a diagram for explaining a case where the object of operation is an operation lever.

[0082] Fig.16 This is a diagram for explaining another example in which the object of operation is an operation lever.

[0083] Fig.17 This is a diagram for explaining a case where a single-leaf door that is opened and closed using a hinged portion as an axis is the object of operation.

[0084] Fig.18 This diagram illustrates a state where a single door is slightly opened by using a smartphone to capture the user's action of pushing his right hand into the air.

[0085] Fig.19 This diagram illustrates a state where a single door has been widely opened by using a smartphone to capture the user's action of pushing his right hand into the air.

[0086] Fig. 20 This is a diagram for explaining a case where a lever-type handle is installed on a single door.

[0087] Fig.21 This is a diagram for explaining a case where a lighting fixture is taken as a subject for imaging.

[0088] Fig. 22 This is a diagram for explaining a case where a switch for controlling lighting of a lighting fixture is taken as a photographic object.

[0089] Fig.23 This is a diagram for explaining a case where an image of a calculator displayed on a display device is used as a photographed object.

[0090] Fig.24 This is a diagram for explaining a situation where the user's actions captured by a smartphone are associated with the operations of operating elements displayed on the screen.

[0091] Fig.25 This is a diagram illustrating an example of a system according to the second embodiment.

[0092] Fig.26 This is a diagram for explaining a situation in which a safe is photographed as an object of operation.

[0093] Fig. 27 This is a diagram for explaining an example of the hardware configuration of a safe.

[0094] Fig.28 This is a flowchart illustrating an example of processing operations executed by a microprocessing unit (MPU).

[0095] Fig.29 This is a diagram illustrating an example of a system according to the fourth embodiment.

[0096] Fig.30 As described in the above embodiment, this shows a situation where the movement of a part of the user's body is accepted as an operation of opening a single door using a smartphone.

[0097] Fig.31 This is a diagram for explaining the structure of information for shared operation by a plurality of users via a network.

[0098] Fig.32 The diagram is a diagram for explaining a situation in which a single door is pushed open using one of three smartphones.

[0099] Fig.33 This is a diagram illustrating how a single door is opened and shared by three smartphones.

[0100] Fig.34 This is a diagram for explaining how to instruct the movement of a potted plant by body movements.

[0101] Fig.35 This is a diagram illustrating the hardware configuration of the robot.

[0102] Fig.36 This is a diagram for explaining the movement of potted plants using a robot.

[0103] Fig.37 It is a diagram for explaining another example of the position facing the image forming apparatus.

[0104] Fig.38 This is a diagram for explaining an example of opening and closing a paper storage section by body movements.

[0105] Fig.39 This is a diagram for explaining an operation example in the case where a mechanism (post-processing device) for performing a punching process or a stapling process on printed paper is installed.

[0106] Fig.40 (A) to (C) in FIG. 1 are diagrams for explaining changes in the screen displayed on the display unit when the image of the post-processing device is selected by the image of the index finger. Fig.40 (A) in the figure shows the display before receiving the body gesture of the image using the index finger. Fig.40 (B) in the figure shows the display after receiving the body gesture of the index finger. Fig.40 (C) in the figure shows an example of a screen of a smartphone that accepts operations using body movements.

[0107] Fig.41 This is a diagram illustrating an example of a function for notifying a user of a situation in which a physical movement of continuing to push a single-door door after it has been opened to the maximum allowable angle has been detected.

[0108] Explanation of symbols

[0109] 100: Image processing device

[0110] 200: Smartphone

[0111] 400: Calculator

[0112] 500: Operation lever

[0113] 600: Single door

[0114] 700: Switch

[0115] 800: Lighting fixtures

[0116] 900: Display device

[0117] 1000, 1300: Network

[0118] 1010: Glasses-type terminal

[0119] 1100: Safe

[0120] 1200: Webcam

[0121] 1400: Server

[0122] 1500: Potted plants

[0123] 1600: Robots DETAILED DESCRIPTION

[0124] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0125] <Implementation method 1>

[0126] <System Example>

[0127] Figure 1 This is a diagram for explaining an example of a system according to the first embodiment.

[0128] Figure 1 The system shown in FIG. 1 includes an image forming apparatus 100 that forms an image on a recording material (hereinafter, sometimes representatively referred to as “paper”) and a smartphone 200 that captures an image of the image forming apparatus 100 .

[0129] Image forming apparatus 100 is an example of an object existing in a real space, and is also an example of an information processing apparatus.

[0130] Image forming apparatus 100 has a copy function for generating a copy, a scanning function for reading an image of an original, a fax transmission and reception function for transmitting fax images to other devices, a printing function for recording images on paper, etc. The copy function is realized by combining the scanning function and the printing function.

[0131] In addition, the image forming device 100 does not need to have the copying function, the scanning function, and the fax transceiver function at the same time, and can also be a device that particularly enhances any one of the functions, such as a copier, a scanner, a fax transceiver, and a printer (including a 3D printer). In addition, the image forming device 100 can also have a device structure that selectively combines some of these functions.

[0132] The image forming apparatus 100 includes a user interface 110 for accepting operations from a user or providing various information to the user; an image reading unit 120 for reading an image of a document; an image forming unit 130 for recording the image on paper; and a storage unit 140 for storing paper.

[0133] The smartphone 200 is a portable terminal device, and as described later, includes a camera for photographing, a display screen, a communication function, etc. The smartphone 200 is an example of an information processing device.

[0134] exist Figure 1 An image 100A of image forming apparatus 100 is displayed on the display screen of smartphone 200 shown in FIG.

[0135] <Configuration of image forming apparatus>

[0136] Figure 2 It is a diagram for explaining an example of the hardware configuration of the image forming apparatus 100 .

[0137] exist Figure 2 In, with Figure 1 The corresponding parts are marked with corresponding symbols.

[0138] The user interface unit 110 includes a display unit 111 for displaying an operation screen and the like, and an operation receiving unit 112 for receiving an input operation by a user.

[0139] The display unit 111 includes, for example, a liquid crystal display panel or an organic electroluminescence (EL) display panel.

[0140] The operation accepting unit 112 includes buttons, a touch panel, and the like for accepting operation inputs by the user.

[0141] The image reading unit 120 includes an image reading section that reads an image from a document and a document conveying section that conveys the document to the image reading section. The image reading unit 120 is disposed above the image forming unit 130 .

[0142] The image forming unit 130 includes an MPU (Micro Processing Unit) 131 that functions as a control unit, a read only memory (ROM) 132 , and a random access memory (RAM) 133 .

[0143] The MPU 131 executes programs such as firmware read from the ROM 132 and provides various functions. The RAM 133 is used as a work area for the programs.

[0144] In addition, the image forming unit 130 includes a storage unit 134 for storing image data, an image processing unit 135 for performing image processing such as color correction or grayscale correction on the image data, an image forming unit 136 for forming an image on paper by electronic photography or inkjet, and a communication unit 137 for communicating with the outside.

[0145] The storage unit 134 has a nonvolatile storage medium such as a hard disk device or a semiconductor memory, and stores image data of a document read by the image reading unit 120 or image data received by the communication unit 137 .

[0146] The image processing unit 135 includes, for example, a dedicated processor or a processing board.

[0147] The communication unit 137 has a local area network (LAN) interface or a fax communication network service interface. Figure 1 ) uses the LAN interface for communication.

[0148] The MPU 131 , ROM 132 , RAM 133 , storage unit 134 , image processing unit 135 , image forming unit 136 , and communication unit 137 are connected to each other via a bus 138 . The user interface unit 110 and the image reading unit 120 are connected to the bus 138 via an interface 139 .

[0149] <Smartphone configuration>

[0150] Figure 3 This is a diagram for explaining an example of the hardware configuration of the smartphone 200 .

[0151] The smart phone 200 includes: a processing circuit unit 210 for processing data; a non-volatile memory 221 for storing programs or data; an audio circuit 222 for inputting and outputting audio signals; a liquid crystal display (LCD) 223 for displaying images; a power control element 224; a camera 225 for taking images; a touch panel 226 for detecting contact with an operating surface; a wireless fidelity (WiFi) module 227 for transmitting and receiving wireless signals in accordance with WiFi (registered trademark) specifications; and a Bluetooth (registered trademark) module 228 for transmitting and receiving wireless signals in accordance with Bluetooth (registered trademark) specifications, which is one of short-range wireless communication specifications.

[0152] In the case of this embodiment, the surface on which the camera 225 is provided and the surface on which the liquid crystal display 223 is provided are in a front-rear relationship.

[0153] Therefore, the user can use the camera 225 to photograph objects (such as an image forming device) and parts of his or her own body (such as fingers, hands, and feet) existing in the real space, and confirm the images using the liquid crystal display 223.

[0154] In addition, when a plurality of cameras 225 or a plurality of liquid crystal displays 223 are provided, at least one set of the camera 225 and the liquid crystal display 223 may be arranged at the front and back surfaces.

[0155] In this embodiment, the liquid crystal display 223 is used, but a display member such as an organic EL (Electro-Luminescence) display may also be used.

[0156] The processing circuit unit 210 is provided with: a memory control unit 211, which controls the reading and writing of data with the memory 221; a plurality of central processing unit cores (CPU cores) 212, which execute processing; a power management unit 213, which manages the supply of power; a system management unit 214, which manages the operation of the embedded system; an audio processing unit 215, which processes audio signals; a graphics processing unit (GPU) 216, which processes images in real time; a display control unit 217, which displays images on a liquid crystal display 223; an external interface (I / F) 218, which is used for connection with external modules; and a baseband circuit 219, which processes baseband signals.

[0157] Figure 4This is a diagram for explaining an example of a functional configuration realized by the processing circuit unit 210 through execution of a program.

[0158] The processing circuit unit 210 here is an example of a detection unit and also an example of a control unit.

[0159] The processing circuit unit 210 functions as a body movement detection unit 251 that processes image data to detect the body movement of the user, an operation object detection unit 252 that detects the object that is the object of operation, an object side function acquisition unit 253 that acquires information on the function of the object from the object that is the object of operation (the image forming device 100 in this embodiment), an operation acceptance unit 254 that accepts the body movement of the user as an operation, an instruction sending unit 255 that sends an instruction corresponding to the accepted operation to the object that is the object, and a reporting unit 256 that reports the acceptance of the operation to the user.

[0160] The body motion detector 251 in this embodiment detects the motion of a part of the user's body (e.g., finger, hand, foot) as a body motion based on the image captured by the camera 225. The meaning of the motion varies depending on the object (e.g., image forming device 100) detected as the object of operation.

[0161] The body movements in this embodiment include, for example, pushing, pulling, turning, grasping, kicking, stepping, clicking, double-clicking, dragging, tapping, flicking, sliding, and the like.

[0162] In this embodiment, the operation object detection unit 252 detects the operation object displayed on the liquid crystal display 223 (see Figure 3 ) to detect the object to be operated or its operating element, etc.

[0163] The operating elements in the present embodiment include physical structures (such as buttons and switches) that are objects of operation, displays (such as buttons and switches) on a screen reproduced by software, and the like.

[0164] The image used for processing is not limited to real-time image data acquired from the camera 225 provided in the smartphone 200 , and may be image data acquired from the memory 221 or from the outside.

[0165] When multiple objects are included in the screen, the operation object detection unit 252 can, for example, detect an object whose image is closer to the center of the screen and is displayed on the LCD display 223 as an object, or can detect an object at a position overlapping with the image of a part of the user's body as an operation object.

[0166] Alternatively, the operation target detection unit 252 may detect a device selected by the user from a group of devices connected to the smartphone 200 via a wireless LAN or the like as an operation target.

[0167] The target-side function acquisition unit 253 in this embodiment acquires information on the operable operating element by, for example, communicating with an object detected by the operation target detection unit 252. For example, the information on the operating element is acquired using a wireless LAN.

[0168] Furthermore, the target-side function acquisition unit 253 may acquire information on operable operating elements from, for example, each device with which communication is possible.

[0169] In addition, the object of operation may not necessarily have a communication function. For example, when the object of operation is a plant or a natural object, it does not have a communication function. In this case, the object-side function acquisition unit 253 may also communicate with a device such as a robot that can perform human actions to obtain information on executable operations.

[0170] The operation accepting unit 254 in this embodiment associates the detected body movement with a specific function of the object to be operated and accepts the association as an operation.

[0171] In addition, if Figure 1 As shown in FIG. 1 , the object to be operated is located at a position facing the user. Therefore, the camera 225 (see FIG. 1 ) Figure 3 ) The direction in which a part of the user's body is photographed becomes the same as the direction in which the object that is the subject of the operation is photographed.

[0172] Here, the so-called facing position means that the user and the object to be operated are facing each other, and it does not necessarily need to be the front. For example, as long as there is a certain relative part between the user (such as the side part or the back part of the object), the object to be operated can be associated with the user's body movement, and instructions can be issued to the associated object. Therefore, the situation where the user is located obliquely in front of the object is also included in the relative situation.

[0173] The instruction transmission unit 255 in this embodiment transmits an instruction to execute the received operation to the target object (eg, image forming apparatus). In addition, the instruction transmission unit 255 also has a function of obtaining a response to the instruction (eg, acceptance of the instruction) from the target object.

[0174] In addition, when each part of the object to be operated is associated with a unique body movement and a functional instruction, the instruction transmitting unit 225 may transmit the instruction received through the body movement. In this case, the operation on the operation screen is not required.

[0175] Furthermore, when an operation on the operation screen is required, it is necessary to select one of the multiple options configured in the operation screen or to reach an option located in a deep hierarchy. However, when specific body movements and specific functions are respectively assigned to specific parts of the object to be operated, instructions can be sent with fewer operations. As a result, multiple functions can be selectively executed quickly.

[0176] The notification unit 256 in this embodiment performs various notifications to assist the operation using the user's body movements through the audio circuit 222 or the liquid crystal display 223 .

[0177] <Processing operations performed by the processing circuit section>

[0178] Figure 5 2 is a flowchart illustrating an example of processing operations performed in the processing circuit unit 210. Figure 5 In the example, the step is represented by the symbol S.

[0179] The processing circuit unit 210 of this embodiment is reflected on the liquid crystal display 223 (see Figure 3 ) is used to identify a part of a human body (such as a finger) in a picture (step 101).

[0180] Next, the processing circuit unit 210 detects the movement of the recognized part of the human body (step 102 ).

[0181] The above processing is performed by, for example, the body movement detection unit 251 (see Figure 4 )implement.

[0182] Then, the processing circuit unit 210 identifies the part of the object that overlaps with the image of a part of the human body in the screen (step 103). This processing is performed by, for example, the operation object detection unit 252 (see Figure 4 )implement.

[0183] In addition, the processing circuit unit 210 also recognizes the object to be operated in advance or simultaneously. When the object to be operated cannot be determined, the processing circuit unit 210 reports that the object to be operated cannot be determined and that an image including the outer shape should be captured. This processing is performed by, for example, the reporting unit 256 (see Figure 4 )implement.

[0184] Then, the processing circuit unit 210 accepts the detected human body movement as an operation on the determined part (step 104). This process is performed, for example, by the operation accepting unit 254. In addition, when there is no function corresponding to the determined part or when there is an action irrelevant to the execution of the function established, the processing circuit unit 210 may not accept the detected movement as an operation.

[0185] Then, the processing circuit unit 210 determines whether the target object can be controlled by communication (step 105). This process is also executed by the operation accepting unit 254, for example.

[0186] When a positive result is obtained in step 105 , the processing circuit unit 210 transmits a signal instructing execution of the operation to the target object (step 106 ).

[0187] When a negative result is obtained in step 105 , the processing circuit unit 210 determines whether there is a self-propelled device that can perform the operation alternatively (step 107 ).

[0188] When a positive result is obtained in step 107 , the processing circuit unit 210 transmits a signal instructing execution of the operation to the self-propelled device (step 108 ).

[0189] When a negative result is obtained in step 107 , the processing circuit unit 210 reports that the operation is not possible (step 109 ).

[0190] <Operation example>

[0191] Hereinafter, an operation example realized by using the above-described functions will be described.

[0192] <Operation example 1>

[0193] Figure 6 This is a diagram for explaining a state in which a user as a photographer takes an image so that his / her index finger 301 overlaps with the image forming apparatus 100 .

[0194] During this photographing, the user holds the smartphone 200 with his left hand. The index finger 301 is the user's right hand and is not in contact with the image forming apparatus 100 .

[0195] exist Figure 6 In this case, a portion of the user interface unit 110 is displayed in an enlarged state on the liquid crystal display 223 of the smartphone 200 .

[0196] This display state is obtained by moving two fingers touching the screen of liquid crystal display 223 in a spreading manner while the image of image forming apparatus 100 is displayed on liquid crystal display 223. In addition, if voice recognition technology is used, it is possible to zoom in even without touching liquid crystal display 223.

[0197] exist Figure 6 In the case of , an image 270 (a button 271 for language switching and a button 272 for adjusting screen brightness) obtained by photographing a portion of the user interface unit 110 existing in the real space and an image 301A of the index finger are displayed on the liquid crystal display 223.

[0198] As described above, the positional relationship in the real space is, from the innermost side, image forming apparatus 100 , user's index finger 301 , and smartphone 200 .

[0199] Therefore, the image 301A of the index finger captured by the smartphone 200 is displayed in a superimposed manner in front of the image 270 of the user interface.

[0200] However, when operating the image forming apparatus 100 by the movement of the index finger 301 , the image forming apparatus 100 must be determined in advance as the object of the operation.

[0201] The image forming apparatus 100 can be identified by, for example, a technique of processing an image of the image forming apparatus 100 to identify the model, a technique of detecting text information such as a model number contained in an image, or a technique of selecting from candidates displayed on the liquid crystal display 223. The candidates here are provided as a group of devices connected to the smartphone 200 via a wireless LAN or the like.

[0202] Here, it is assumed that image forming apparatus 100 has been determined as the target of operation.

[0203] exist Figure 6 In the case of, two operable operating elements (a button 271 for language switching and a button 272 for screen brightness adjustment) are displayed on the liquid crystal display 223, but three or more operating elements may be displayed, or only one operating element may be displayed.

[0204] exist Figure 6 In the case of , since two operating elements are displayed on the liquid crystal display 223, the index finger 301 must be moved in the air in such a way that the operating element to be operated and the image 301A of the index finger overlap.

[0205] In the smartphone 200, the movement of the index finger 301 in the air is also detected as a user's motion. However, the motion in the horizontal direction is not associated with the motion of operating the operating element being captured.

[0206] Therefore, the action before the image 301A of the index finger overlaps the button 272 for adjusting the screen brightness is not accepted as an operation on the operating element.

[0207] In addition, in this embodiment, since the object of operation is a button, the movement of the index finger image 301A is detected, but a part of the body to be detected can also be determined according to the object of operation. For example, a part of the human body such as a hand, an arm, or a foot can also be used as the object of detection. By determining the object of detection, erroneous operation can be avoided.

[0208] Figure 7 3 is a diagram for explaining how an image is captured of an action such as pressing the index finger 301 in the air without contacting the image forming apparatus 100. Figure 7 In, with Figure 6 The corresponding parts are indicated by corresponding symbols.

[0209] The movement of pressing the index finger 301 here refers to a movement in a direction in which the index finger 301 is separated from the smartphone 200 . In other words, it refers to a movement in a direction in which the index finger 301 is moved away from the smartphone 200 .

[0210] Specifically, in the real space, the same action as pressing the operating element displayed on the user interface unit 110 is performed. This is because the direction of the user's line of sight is the same as that of the camera 225 (see FIG. 21 ). Figure 3 )’s shooting direction is consistent.

[0211] exist Figure 7 In the liquid crystal display 223 shown in FIG. , since the image 301A of the index finger overlaps the button 272 for adjusting the screen brightness, the smartphone 200 detects the movement of the index finger 301 as an operation of pressing the button 272 for adjusting the screen brightness.

[0212] In the case of the present embodiment, in the smartphone 200, the relationship between the operating elements and the actions is stored in the memory 221 (see Figure 3 However, the relationship stored in the memory 221 may also be about a part of the operating elements provided in the image forming apparatus 100.

[0213] Furthermore, the relationship between the image of the operating element provided in image forming apparatus 100 and the motion recognized as the operation is stored in image forming apparatus 100, and image forming apparatus 100 may execute processing for specifying the operating element to be operated.

[0214] exist Figure 7Since the smartphone 200 can communicate with the image forming apparatus 100 , the smartphone 200 sends a signal indicating the operation of the button 272 for adjusting the screen brightness to the image forming apparatus 100 .

[0215] The smartphone 200 in the present embodiment is provided with a function of notifying the user of the acceptance of the operation.

[0216] Therefore, in Figure 7 In the process, the display form of the button 272 for adjusting the screen brightness is changed to notify the user of the acceptance of the operation.

[0217] Here, as a method of changing the display form, for example, a method of increasing the brightness of the area where the button to be operated is displayed, causing it to blink, or changing it to another color can be used.

[0218] In addition, the function of reporting the acceptance of the operation to the user is not an essential function.

[0219] <Operation example 2>

[0220] Figure 8 This is a diagram for explaining another method of reporting the acceptance of an operation to the user.

[0221] exist Figure 8 In, with Figure 7 The corresponding parts are marked with corresponding symbols.

[0222] exist Figure 8 In this case, although there is no change in the display of the button 272 for adjusting the screen brightness, a "click" sound is output from the speaker not shown in the figure.

[0223] By generating this sound, the user can confirm that his or her action has been accepted as an operation.

[0224] Alternatively, the notification method using sound may be in the form of words such as "operation accepted".

[0225] <Operation example 3>

[0226] Fig. 9 This is a diagram for explaining another method of reporting the acceptance of an operation to the user.

[0227] exist Fig. 9 In, with Figure 7 The corresponding parts are marked with corresponding symbols.

[0228] exist Fig. 9 In the case of , although there is no change in the display of the button 272 for adjusting the screen brightness, a small screen 280 indicating acceptance of the operation is additionally displayed.

[0229] exist Fig. 9 In the screen 280 shown in FIG. 1 , a character string "Operation accepted" is described. Of course, this character string is an example.

[0230] <Operation example 4>

[0231] Fig.10 This is a diagram for explaining another method of reporting the acceptance of an operation to the user.

[0232] exist Fig.10 In, with Figure 7 The corresponding parts are marked with corresponding symbols.

[0233] exist Fig.10 In the case of , although there is no change in the display of the button 272 for adjusting the screen brightness, the form of the display of the image 301A of the index finger is changed from the form before the operation is accepted.

[0234] Here, as a method of changing the displayed form, for example, a method of increasing the brightness of the area where the index finger image 301A is displayed, making it blink, or changing it to another color may be used.

[0235] <Operation example 5>

[0236] Fig.11 This is an illustration of not using the index finger as image 301A (refer to Figure 7 ) is a diagram of a method displayed in the LCD display 223.

[0237] exist Fig.11 In, with Figure 7 The corresponding parts are marked with corresponding symbols.

[0238] For example, when the intrusion to the camera 225 (see Figure 3 ), there is also a possibility that most of the operating element is blocked by a part of the human body. For example, there is a possibility that the text written on the operating element is blocked by the image of the hand or arm.

[0239] In such a case, the user cannot confirm the operating element to be operated on the screen, or can only confirm a part of it, which may cause trouble in operation.

[0240] If the characters written on the operating element cannot be confirmed, not only will it be difficult to know which operating element is being operated, but it may also cause erroneous operation.

[0241] Therefore, in Fig.11 In the example, the image 301A of the index finger is removed from the screen by image processing. In this case, the user can confirm which operating element is to be operated on the screen during the operation.

[0242] <Operation Example 6>

[0243] On the other hand, in the case of operation example 5 (refer to Fig.11 ) is not displayed, and thus it is difficult to know the relationship between the positions of the operating element and the image 301A of the index finger, or it may be difficult to know whether the operation has actually been accepted.

[0244] Fig.12 This is an illustration of not using the index finger as image 301A (refer to Figure 7 ) is a diagram of another method displayed in the LCD display 223.

[0245] exist Fig.12 In, with Fig.11 The corresponding parts are marked with corresponding symbols.

[0246] exist Fig.12 In the example, the display form of the operating element (here, the button 272 for adjusting the screen brightness) at the position where the index finger image 301A overlaps is actually changed to support the user's confirmation. Fig.12 In the example, a screen 280 is also displayed in which a character string is used to record that an operation has been accepted.

[0247] This display allows the user to operate a specific operating element without being blocked by a part of the user's body. In addition, if the user has a function to change the display form of the operating element, the user only needs to avoid mistaking the object of the operation. Furthermore, if the acceptance of the operation is reported, the user's operability can be improved.

[0248] <Operation Example 7>

[0249] Fig.13 This is an illustration of not using the index finger as image 301A (refer to Figure 7 ) is a diagram of another method displayed in the LCD display 223.

[0250] exist Fig.13 In, with Fig.11 The corresponding parts are marked with corresponding symbols.

[0251] In operation example 6, although the image 301A of the index finger is not displayed at all, it can be imagined that the user wants to confirm the position on the screen.

[0252] Fig.13 This is a method of displaying a desired area in consideration of this, and displays the outline of the area where the image 301A of the index finger is located. By displaying the outline, the user can easily confirm the relationship between the position of the index finger 301 and the position of the operating element. The outline here is an example of a representation suggesting a part of the user's body.

[0253] In addition, Fig.13 In FIG. 1 , the outline of the image 301A of the index finger is indicated by a dotted line, but an image converted into a transmission image by image processing may be displayed instead of a real image. The transmission image here is also an example of a representation suggesting a part of the user's body.

[0254] In addition, only the fingertip portion can be selectively expressed by an outline or a transparent image, and the rest of the fingertip can be displayed by a real image, or vice versa, only the fingertip portion can be selectively displayed by a real image. Since the area blocked by the image of a part of the human body is reduced, it is easy to confirm the image of the object to be operated.

[0255] <Operation example 8>

[0256] Fig.14 This is a diagram for explaining a case where the object of operation is a calculator 400 having a communication function.

[0257] In this example, even if the fingertips of the user's right hand are dirty, the user can operate the calculator 400 without making it dirty and check the result of the calculation.

[0258] exist Fig.14 In the case of , the image 301A of the user's index finger is displayed as a transmissive image superimposed on the image 400A of the calculator. The transmissive image is different from the outline and is displayed transparently like a real image.

[0259] However, when the size of the operating element is small as in the calculator 400, there is a possibility of pressing the wrong button.

[0260] Therefore, the size of the part of the body displayed on the liquid crystal display 223 may be selectable. For example, it may be made larger or smaller than the real image. In addition, it may be made adjustable in stages.

[0261] By providing this function, it is expected that the number of erroneous pressing of the operating element will be reduced.

[0262] <Operation example 9>

[0263] Fig.15 This is a diagram for explaining a case where the object of operation is the operation lever 500 .

[0264] The operation lever 500 in the present embodiment includes, for example, a joystick, a control lever, a shift lever, a direction indicator lever, a brake lever, and a controller of the vehicle.

[0265] exist Fig.15In the case of, since the object of operation is the operating rod 500, the user extends the right hand 310 in the air as if holding something. In this case, if the right hand 310 is moved forward, backward, left, and right, the operating rod 500 performs the same action through the instruction from the smart phone 200 that detects this action.

[0266] The right hand image 310A is displayed as if the image 500A holding the joystick is on the liquid crystal display 223. Alternatively, the right hand image 310A may be displayed simply superimposed on the joystick image 500A.

[0267] However, the operation stick 500 does not need to be moved in conjunction with the right hand 310 , and it is sufficient that the object controlled by the operation stick 500 is controlled in accordance with the movement of the right hand 310 .

[0268] The reason is that the operating rod 500 is an input device and not the final object of control. Therefore, communication between the operating rod 500 and the smartphone 200 is not necessary. Even if communication with the operating rod 500 is impossible, communication between the smartphone 200 and the device controlled by the operation of the operating rod 500 is sufficient.

[0269] The smartphone 200 in this operation example has a function of decorating a part of the user's body displayed on the liquid crystal display 223 according to the user's attributes or the environment in which the user is used.

[0270] For example, in Operation Example 1 (refer to Figure 6 ) in the case of camera 225 (refer to Figure 3 ) The part of the body photographed is directly displayed as a real image, but in Fig.15 In the example of FIG. 3 , an image 310A of a right hand wearing a thin glove is displayed. The glove here is an example of equipment.

[0271] The type of decoration to be used may be selected by the user in advance, or may be notified from the operating lever 500 (or a device having the operating lever 500), or may be stored in the memory 221 of the smartphone 200 (see Figure 3 ) can also be notified from an external server, etc.

[0272] In addition, the image data used for decoration may be stored in the memory 221 of the smartphone 200, or may be provided from a device including the operation stick 500, an external server, or the like.

[0273] The shape and size of the right hand image 310A displayed on the liquid crystal display 223 also vary depending on the shooting method. Therefore, the smartphone 200 also has a function of deforming the decorative image according to the shape and size of the right hand image 310A.

[0274] In the present embodiment, this function is realized by executing an application in the smartphone 200. However, the image processing may be executed by a computer such as a server provided outside the smartphone 200, and the result of the processing may be reflected in the display of the liquid crystal display 223.

[0275] Fig.16 This is a diagram for explaining another example of a case where the object of operation is the operation lever 500 .

[0276] exist Fig.16 In, with Fig.15 The corresponding parts are marked with corresponding symbols.

[0277] exist Fig.16 In the case of , it is assumed that the operating lever 500 is operated outdoors in winter. Fig.16 In this case, a thick glove is attached to the image 310A of the right hand as a decoration.

[0278] By displaying the decorated right hand image 310A on the liquid crystal display 223 in this manner, the user can be given the same impression as when actually operating the operating lever 500 , thereby enhancing the sense of reality of the operation.

[0279] <Operation example 10>

[0280] Fig.17 This is a diagram for explaining a case where a single-opening door 600 that is opened and closed using a hinged portion as an axis is an object of operation.

[0281] The single door 600 is provided with a mechanism 610 for electrically opening or closing the single door 600. The mechanism 610 is installed on the wall surface side where the entrance and exit are provided. In addition, the mechanism 610 here can communicate directly or indirectly with the smartphone 200 via a wireless LAN or the like.

[0282] In addition, a switch 700 for turning on or off a lighting device (not shown) is mounted on a wall surface near the entrance.

[0283] Therefore, the image 600A of the single door and the image 700A of the switch are displayed on the liquid crystal display 223 of the smartphone 200 .

[0284] In addition, the user's right hand 310 is located on the line connecting the single door 600 existing in the real space and the smartphone 200. The right hand 310 is not in contact with the single door 600.

[0285] Therefore, on the liquid crystal display 223, the right-hand image 310A is displayed as if overlapping with the single-door image 600A. Fig.17 In this case, the image 310A of the right hand is a real image.

[0286] Fig.17 The single door 600 shown in the figure is opened toward the inside by pushing and is closed by pulling toward the front.

[0287] Fig.18 This figure illustrates a state in which the single door 600 is slightly opened by photographing the user's action of pushing the right hand 310 into the air using the smartphone 200.

[0288] exist Fig.18 In FIG. 6 , the direction of the movement of the right hand 310 is indicated by an arrow. In the smartphone 200, the movement of the right hand 310 is detected as an operation of opening the single door 600. The smartphone 200 transmits a signal instructing the execution of the detected operation to the mechanism 610.

[0289] At this time, the smartphone 200 may also send the speed of the movement of the right hand 310 detected in the image to the mechanism 610. In this case, the mechanism 610 may reflect the speed of opening the single door 600 in accordance with the provided speed. That is, the single door 600 can be opened with a feeling close to the actual operation of the single door 600.

[0290] Fig.19 This diagram illustrates a state in which the single door 600 is largely opened by photographing the user's action of pushing the right hand 310 into the air using the smartphone 200.

[0291] exist Fig.19 In the embodiment, the movement of the right hand 310 continues, and the drive by the mechanism 610 also continues.

[0292] In addition, after the action of opening the single door 600 is detected, it is possible to control the single door 600 to continue to be driven even if the action of the right hand 310 stops. The reason for this is that during the shooting process using the smartphone 200, the range in which the right hand 310 can move is narrow relative to the swing range of the single door 600 that is the object of driving. This control can be implemented as a function of the smartphone 200 or as a function of the mechanism 610.

[0293] Furthermore, when the action of pulling the right hand 310 forward is detected, the smartphone 200 detects this action as an operation of closing the single door 600 .

[0294] Fig. 20 The diagram is for explaining a case where a lever-type handle (rod-shaped handle) 620 is attached to a single-leaf door 600 .

[0295] When opening the single door 600, it is necessary to grasp the lever-like handle 620, rotate it, and then push it out.

[0296] exist Fig. 20 In the figure, the image 310A of the right hand is displayed superimposed on the image 600A of the single-opening door and the image 620A of the handle.

[0297] When the two-stage actions are detected in sequence, Fig. 20 The smartphone 200 shown in FIG. detects the two-stage action as an operation of opening the single door 600. Fig.18 and Fig.19 Same description as .

[0298] In addition, Figures 17 to 20 A single-leaf door 600 is illustrated in the figure, but it is not limited to a hinged door that opens and closes by drawing an arc on the door surface. It can also be a sliding door that opens and closes in a straight line guided by a groove or a guide rail, a glide slide door that combines a hinged door and a sliding door, a folding door, a revolving door, etc.

[0299] In addition, the door is not limited to being single-opening, but may be double-opening.

[0300] <Operation Example 11>

[0301] Fig.21 This is a diagram for explaining a case where the lighting fixture 800 is a subject to be photographed.

[0302] Fig.21 The lighting fixture 800 shown in FIG. 8 is electrically connected to the position-holding switch 700 through wiring (not shown). When the switch 700 is turned on, the lighting fixture 800 is turned on, and when the switch 700 is turned off, the lighting fixture 800 is turned off.

[0303] exist Fig.21 In this case, the lighting fixture 800 is provided with the following function: the action of pushing the index finger 301 toward the air when the light is off is accepted as a lighting operation, and the action of pushing the index finger 301 toward the air when the light is on is accepted as a lighting operation.

[0304] Therefore, the image 301A of the index finger is superimposed on the image 800A of the lighting fixture displayed on the liquid crystal display 223 of the smartphone 200, and when the index finger 301 is pushed into the air, the smartphone 200 that detects this action outputs a signal for switching the state to the lighting fixture 800. Specifically, if the lighting fixture 800 is in the lighting state, a signal for turning off the lighting is output, and if the lighting fixture 800 is in the lighting state, a signal for turning on the lighting is output.

[0305] In addition, the switching of the state of the lighting device 800 is not limited to two stages. For example, even in the state of lighting, the state of several stages with different brightness can be switched by the user's action. In addition, if the lighting device 800 can switch the color of the lighting light, the color can also be switched by the user's action.

[0306] In the above description, the action of pushing the index finger 301 in the air is detected as an operation, but one finger may be detected as turning on the light, two fingers may be detected as turning off the light, or a finger raised may be detected as turning on the light, and a finger placed flat may be detected as turning off the light. Figure 3 )The number of fingers photographed or the direction of the fingers is associated with a specific operation.

[0307] In short, the operation state of the lighting fixture 800 can be operated without directly contacting the lighting fixture 800 .

[0308] Fig. 22 This is a diagram for explaining a case where a switch 700 for controlling lighting of a lighting fixture 800 is taken as a subject for imaging.

[0309] exist Fig. 22 In, with Fig.21 The corresponding parts are marked with corresponding symbols.

[0310] As described above, the lighting fixture 800 is turned on by the on operation of the switch 700 , and is turned off by the off operation of the switch 700 .

[0311] In addition, Fig. 22 In the case of selectively displaying only the tip of the index finger as image 301A, and not displaying other parts, the same as Fig.21 By displaying only the parts related to the operation as described above, it becomes easy for the user to confirm the object of the operation performed.

[0312] When the switch 700 has a function of communicating with the smartphone 200, if there is an operation of pushing the index finger 301 into the air with the image 301A of the index finger superimposed on the image 700A of the switch, the smartphone 200 outputs a signal to switch the switch 700 on and off.

[0313] In addition, there is a case where the switch 700 does not have a function of communicating with the smartphone 200. In this case, as long as the relationship between the switch 700 and the lighting fixture 800 is provided to the smartphone 200, the action of pushing the index finger 301 in the air while the image 301A of the index finger is superimposed on the image 700A of the switch can be accepted as an operation on the lighting fixture 800.

[0314] In addition, the objects to be controlled may be other items such as air conditioners, audio equipment, and home electrical appliances.

[0315] <Operation Example 12>

[0316] In the above operation example, an object existing in a real space is photographed using the smartphone 200, but the object to be photographed may also be an image of an actually existing object.

[0317] Fig.23 This is a diagram for explaining a case where an image 400A of a calculator displayed on the display device 900 is used as a photographic object.

[0318] exist Fig.23 In, with Fig.14 The corresponding parts are marked with corresponding symbols.

[0319] The display device 900 is, for example, a television receiver or a monitor. The image 400A of the computer here is an image reproduced from image data obtained by photographing the actual computer 400.

[0320] Here, the image 400B of a calculator and the image 301A of an index finger, which are obtained by photographing the display device 900 , are displayed on the liquid crystal display 223 of the smartphone 200 . Of course, the index finger 301 is not in contact with the display device 900 .

[0321] In this operation example, except that the object photographed by the smartphone 200 is different from the actual calculator 400 that is the object of operation, the operation is similar to that performed by the user. Fig.14 The operation example 8 described is the same.

[0322] In this case, the operation of the calculator 400 can be performed in the same manner as in Operation Example 8 as long as the calculator 400 to be operated is previously determined from a group of devices connected to the smartphone 200 via a wireless LAN or the like.

[0323] In addition, even if the relationship between the image 400A of the calculator and the actual calculator 400 is unclear, the smartphone 200 can detect that the photographed object is a calculator by using the image recognition technology. In this case, the smartphone 200 can also specify the actual calculator 400 with which communication can be performed and instruct the execution of the operation corresponding to the user's action.

[0324] However, if the calculator 400 that can communicate is not prepared for an operation that can be detected by the smartphone 200 , the calculator 400 cannot be operated by capturing a body movement.

[0325] <Operation Example 13>

[0326] In the above operation example, it is assumed that the camera 225 (see Figure 3 ) The object that is the object of operation (or its image) and a part of the user's body are photographed at the same time, but the object that is the object of operation does not need to be photographed.

[0327] Fig.24 This is a diagram for explaining a case where the user's actions captured by the smartphone 200 are associated with the operations of the operating elements displayed on the screen.

[0328] exist Fig.24 In, with Figure 6 The corresponding parts are marked with corresponding symbols.

[0329] exist Fig.24 In this case, the image forming apparatus 100 as the operation target is connected to the smartphone 200 via a network 1000 such as the Internet or a LAN so as to be communicable.

[0330] exist Fig.24 In the case of , an image 270 (a button 271 for switching the language and a button 272 for adjusting the screen brightness) of a portion of the user interface unit 110 is displayed on the liquid crystal display 223 .

[0331] Although image forming apparatus 100 is not present in front of smartphone 200 , a composite image of image 301A of the index finger and image 270 captured by camera 225 is displayed on liquid crystal display 223 .

[0332] When the smartphone 200 takes the synthesized image as a target for image processing and accepts the movement of the index finger 301 as an operation, it transmits a signal instructing execution of the operation to the corresponding image forming apparatus 100 .

[0333] <Implementation method 2>

[0334] In the case of the first embodiment, the smartphone 200 (see Figure 1 ) to capture the user's body movements and use them in an actual image forming device 100 (refer to Figure 1 ) operation, but the capture of the user's body movements is not limited to the smart phone 200.

[0335] Fig.25 This is a diagram illustrating an example of a system according to the second embodiment.

[0336] exist Fig.25 In, with Figure 1 The corresponding parts are marked with corresponding symbols.

[0337] exist Fig.25In the case of , the user wears the glasses-type terminal 1010. The hardware configuration of the glasses-type terminal 1010 (see Figure 3 ) or software composition (refer to Figure 4 ) is the same as the smartphone 200. However, the glasses-type terminal 1010 does not have a touch panel 226 (see Figure 3 ).

[0338] In the case of the glasses-type terminal 1010, the user can freely use both hands. Therefore, the glasses-type terminal 1010 can be used to input operations using both hands. For example, it can be used to instruct operations for a humanoid robot. Therefore, compared with the case of using the smartphone 200, the range of inputtable operations can be expanded.

[0339] in addition, Fig.25 The user can see the image 110A of the user interface.

[0340] <Implementation method 3>

[0341] In the above-described embodiments, the devices to be operated (eg, image processing apparatus, calculator, joystick, door, lighting fixture) are the same in that no distinction is made between physical operation and operation using body movements.

[0342] In the third embodiment, a device that can limit the acceptance of operations to operations using body movements will be described.

[0343] Fig.26 This is a diagram for explaining a situation in which the safe 1100 is photographed as an object of operation.

[0344] exist Fig.26 In the safe 1100 shown in FIG. 1 , in addition to the function of validating unlocking on the condition that the cylinder 1101 or the button 1102 arranged at the front is correctly operated physically, it also has the function of invalidating these physical operations.

[0345] When the physical operation is invalidated, unlocking is enabled when a signal corresponding to the operation of the button 1102 is received from the smartphone 200 .

[0346] In this embodiment, the image of the safe 1100A and the image of the index finger 301A are also displayed on the liquid crystal display 223. Of course, the image of the index finger 301A corresponds to the index finger 301. The index finger 301 does not touch the safe 1100, but moves in the air.

[0347] Fig. 27 This is a diagram for explaining an example of the hardware configuration of the safe 1100 .

[0348] The safe 1100 includes: a lock cylinder 1101; a plurality of buttons 1102, with alphanumeric characters printed on the operation surface; an MPU 1111; a ROM 1112 for storing data such as firmware; a RAM 1113 for use as a program operation area; a communication unit 1114 for communicating with a smartphone 200 (see Fig.26 ) and communication with external devices such as the unlocking control unit 1115, which electrically unlocks or locks the lock cylinder 1101.

[0349] Fig.28 This is a flowchart illustrating an example of processing operations executed by the MPU 1111.

[0350] First, the MPU 1111 determines whether the physical operation has been invalidated (step 201 ).

[0351] When a positive result is obtained in step 201 , the MPU 1111 validates the setting to accept only the operation from the smartphone 200 (step 202 ).

[0352] On the other hand, when a negative result is obtained in step 201, the MPU 1111 validates the setting for accepting physical operations (step 203). When the setting for accepting physical operations is valid, the operation from the smartphone 200 is also valid.

[0353] After these settings, the MPU 1111 determines whether the content of the operation meets the unlocking conditions (step 204).

[0354] When a positive result is obtained in step 204 , the MPU 1111 unlocks the key cylinder 1101 (step 205 ).

[0355] On the other hand, when a negative result is obtained in step 204 , the MPU 1111 invalidates the operation (step 206 ).

[0356] For example, in the case where the physical operation has been invalidated, whether the correct key is inserted into the lock cylinder 1101 (refer to Fig.26 ) or operate buttons 1102 in the correct order (refer to Fig.26 ), all maintain the locked state. On the other hand, when an instruction to operate the buttons 1102 in the correct order is provided from the smartphone 200, unlocking is permitted.

[0357] In the safe 1100 of the present embodiment, the setting that cannot be unlocked even if the physical operation is correct is set to be effective, thereby improving the anti-theft performance. In addition, by combining the function of personal authentication using the smart phone 200, the user who can unlock the safe 1100 is limited to a specific individual (a user who can operate the smart phone 200).

[0358] <Implementation method 4>

[0359] Here, a method of operating by body movements as if a subject photographed by a web camera or the like is in front of one's eyes will be described.

[0360] Fig.29 This is a diagram illustrating an example of a system according to the fourth embodiment.

[0361] exist Fig.29 In the system shown in , image data output from the network camera 1200 is transmitted to the smartphone 200 via the network 1300 and is displayed on the liquid crystal display 223 .

[0362] exist Fig.29 An image 600A obtained by photographing the single door 600 is displayed on the liquid crystal display 223 shown in FIG.

[0363] Fig.30 As described in the above embodiment, the case where the movement of a part of the user's body (here, the right hand 310 ) is accepted as the operation of the single door 600 by the smartphone 200 .

[0364] exist Fig.30 In, with Fig.29 The corresponding parts are marked with corresponding symbols.

[0365] exist Fig.30 In the figure, the image 310A of the right hand taken by the smartphone 200 is superimposed on the image 600A of the single door.

[0366] In this state, if the right hand 310 is moved as if pushing out in the air, a signal instructing the opening operation of the single door 600 is output to the mechanism 610 connected via the network 1300 .

[0367] Fig.31 1300 is a diagram for explaining the structure of information shared by multiple users via the network 1300. Fig.31 In, with Fig.30 The corresponding parts are marked with corresponding symbols.

[0368] exist Fig.31 , it is shown that the server 1400 obtains the image data output from the network camera 1200 and then sends it to the three smartphones 200. In addition, the function of the server 1400 may also be built in the network camera 1200.

[0369] exist Fig.31 In this case, an image 600A of a single door is displayed on any of the smart phones 200.

[0370] Fig.32 2 is a diagram for explaining a situation where a single door 600 is pushed open using one of the three smartphones 200. Fig.32 In, with Fig.31 The corresponding parts are marked with corresponding symbols.

[0371] exist Fig.32 In the case of a situation where the operation received by one of the three smartphones 200 is sent to the mechanism 610 of the single door 600 through the server 1400. In addition, in the case where multiple operations on the single door 600 are generated at the same time, the server 1400 in this embodiment gives priority to the operation that arrives first. In addition, in the case where conflicting operations arrive at the same time, the execution can also be reserved. The so-called "simultaneous" means arriving within a pre-set interval.

[0372] Fig.33 This is a diagram for explaining how the single door 600 is opened and shared by three smartphones 200. Fig.33 In, with Fig.32 The corresponding parts are marked with corresponding symbols.

[0373] If this structure is used, multiple people can operate one device through their own body movements.

[0374] <Implementation method 5>

[0375] In the above embodiment, a case where the operation target displayed on the smartphone 200 is a device is described. However, here, a case where the communication function is not provided in the operation target is described.

[0376] Fig.34 1 is a diagram for explaining how to instruct the movement of the potted plant 1500 by body movements. Fig.34 In FIG. 1 , an image 310A of the user's right hand and an image 1500A of a potted plant are displayed on the screen of the smartphone 200. In this case, the right hand 310 in the real space is also not in contact with the potted plant 1500.

[0377] The potted plant 1500 has neither a communication function nor a self-propelled device installed therein. Therefore, even if the movement of the right hand 310 is photographed by the smartphone 200 as in the above embodiment, the potted plant 1500 cannot be moved by the movement.

[0378] In this embodiment, a humanoid robot 1600 having a self-propelled function is designated as a transmission target of the operation using body movements. In addition, the robot 1600 as the transmission target of the instruction can be designated from a group of devices that can communicate with the smartphone 200, for example.

[0379] Fig.34Although the robot 1600 shown in the figure adopts a humanoid appearance, the appearance can be arbitrary as long as the necessary functions are installed. For example, it can also imitate animals such as dogs and cats, plants such as flowers and trees, and vehicles such as cars (including trains) or airplanes.

[0380] Fig.34 The robot 1600 shown in FIG. 1 has a torso 1611 , a head 1612 , arms 1613 , arms 1615 , hands 1614 , hands 1616 , and feet 1617 , feet 1618 .

[0381] Electronic components for signal processing are stored in the body 1611. The body 1611 may also be equipped with a display element or an audio device.

[0382] The head 1612 is connected to the trunk 1611 via a joint mechanism provided at the neck. In the present embodiment, the joint mechanism can rotate around three axes. The so-called rotation around three axes refers to yaw (rotation around the z-axis), roll (rotation around the x-axis), and pitch (rotation around the y-axis). The joint mechanism here is an example of a movable part.

[0383] It is not necessary to be rotatable about all three axes, and it may be rotatable about only one axis, or about two axes. These rotations may be achieved by a motor not shown, but may also be performed manually. However, this does not prevent the head 1612 from being fixed relative to the trunk 1611.

[0384] Eyes 1612A and 1612B are provided on the head 1612. The eyes 1612A and 1612B may be provided as decorations, or may be equipped with a camera, a projector, a lighting device, etc. The head 1612 may also be provided with movable ears.

[0385] The arm 1613 and the arm 1615 in this embodiment are connected to the joint mechanism via the trunk 1611. The upper arm and the forearm of the arm 1613 and the arm 1615 are connected via the joint mechanism. Similar to the head 1612, the joint mechanism here can be multi-axis or single-axis. In addition, the rotation around the axis can be achieved by a motor not shown in the figure, or it can be achieved manually. However, the arm 1613 and the arm 1615 can also be fixed to the trunk 1611. The joint mechanism here is also an example of a movable part.

[0386] In addition, if the arms 1613 and 1615 are bent to a predetermined angle, they can be used to transport objects.

[0387] Hand 1614 and hand 1616 are connected to arm 1613 and arm 1615 via a joint mechanism provided at the wrist. The palms and fingers of hand 1614 and hand 1616 are connected via a joint mechanism. Similar to head 1612, the joint mechanism here can be multi-axis or single-axis. In addition, the rotation around the axis can be achieved by a motor not shown in the figure, or it can be achieved manually. In the case of this embodiment, hand 1614 and hand 1616 can grasp an object by opening and closing the fingers. The joint mechanism here is also an example of a movable part.

[0388] Hands 1614 and 1616 may also be fixed relative to arms 1613 and 1615 .

[0389] The legs 1617 and 1618 may be connected to the trunk 1611 via a joint mechanism, or may be installed on the trunk 1611 as a self-propelled device such as wheels or tracks.

[0390] When the legs 1617 and 1618 are connected to the trunk 1611 via a joint mechanism, the joint mechanism may be multi-axial or uniaxial, similar to the head 1612 .

[0391] In addition, the rotation around the axis can be realized by a motor not shown in the figure, or can be realized manually. In addition, the legs 1617 and 1618 can also be fixed relative to the trunk 1611. The joint mechanism here is also an example of a movable part.

[0392] Fig.35 This is a diagram for explaining the hardware configuration of the robot 1600 .

[0393] The robot 1600 includes: a control unit 1621 for controlling the overall movement of the device; a camera 1622 for capturing images of the robot's surroundings; a speaker 1623 for reproducing voice, music, and sound effects for conversation; a microphone 1624 for inputting or acquiring sounds; a movable mechanism 1625 such as a joint mechanism; a communication unit 1626 for communicating with an external device (e.g., a smartphone); a display unit 1627 for displaying images; a moving mechanism 1628 for moving the device as a whole; a power supply 1629 for supplying power to each unit; a sensor 1630 for collecting the status of each unit or surrounding information; and a position detection unit 1631 for acquiring position information. The above-mentioned units are connected to each other, for example, via a bus 1632.

[0394] certainly, Fig.35 The hardware configuration shown in is an example. Therefore, the robot 1600 does not need to be equipped with all the functional parts described above.

[0395] In addition, the robot 1600 may further be equipped with functional parts not shown in the figure. For example, the robot 1600 may be equipped with a power button, a storage device (hard disk device, semiconductor memory, etc.), a heat source (including a cooling source), etc.

[0396] The control unit 1621 is a so-called computer, and has a CPU, a ROM, and a RAM. The ROM stores a program to be executed by the CPU. The CPU reads the program stored in the ROM and uses the RAM in the work area to execute the program. Through the execution of the program, the CPU controls the actions of the various parts constituting the robot 1600.

[0397] The program here includes a program related to the implementation of an algorithm corresponding to artificial intelligence. The CPU and RAM constituting the control unit 1621 provide computing resources used by artificial intelligence.

[0398] The control unit 1621 in this embodiment processes the information obtained through the camera 1622, the microphone 1624, and the sensor 1630 through, for example, artificial intelligence, and autonomously determines the action corresponding to the surrounding environment or state of the robot 1600.

[0399] For example, voice may be output through the speaker 1623 , a message may be sent through the communication unit 1626 , and an image may be output through the display unit 1627 .

[0400] The control unit 1621 can communicate with the user through the input and output of such information or the operation of the movable mechanism 1625. Examples of application of communication include, for example, meeting guests or conducting conferences.

[0401] The control unit 1621 is also equipped with a function of collecting additional information by searching on the Internet or communicating with an external computer when an unknown situation occurs, and finding a solution based on the similarity with the search items.

[0402] In the case of this embodiment, the information acquired by the control unit 1621 includes, for example, vision, hearing, touch, taste, smell, sense of balance, and temperature.

[0403] Vision is achieved through recognition processing of images captured by camera 1622.

[0404] The sense of hearing is achieved by recognition processing of the sound acquired by the microphone 1624 .

[0405] The sense of touch includes, for example, surface sensation (touch, pain, temperature), deep sensation (pressure, position, vibration, etc.), and cortical sensation (two-point discrimination, stereoscopic discrimination, etc.).

[0406] The control unit 1621 may distinguish the difference in tactile sensation.

[0407] The sense of touch, taste, smell, balance, and temperature are realized by detecting information using various sensors 1630. In addition, temperature also includes ambient temperature, internal temperature, body temperature of a person or an animal, and the like.

[0408] Furthermore, the information acquired by the control unit 1621 may include the brain waves of a person or an animal. In this case, the communication unit 1626 may receive the information transmitted from the brain wave detection element worn by the person or the like.

[0409] In the present embodiment, the camera 1622 is disposed between the eye 1612A and the eye 1612B (see Fig.34 ) position.

[0410] When a projector is used as the display unit 1627, the projector can be placed, for example, between the eye 1612A and the eye 1612B (see Figure 1 ) on either or both of the torso 1611 and the head 1612. In addition, the projector may also be disposed on the torso 1611 or the head 1612.

[0411] In addition to conveying objects, the movable mechanism 1625 can also be used for the purpose of expressing emotions.

[0412] When used for transporting an object, the movable mechanism 1625 is moved by the arm 1613, the arm 1615 or the hand 1614, the hand 1616 (see Fig.34 ) to achieve actions such as grasping objects, hugging objects, and supporting objects.

[0413] When used to express emotions, the movable mechanism 1625 may be, for example, the head 1612, the arm 1613, the arm 1615, the hand 1614, the hand 1616, etc. (see Fig.34 ) to perform actions such as tilting the head, looking up, looking around (looking around), cheering, pointing with the hands, etc.

[0414] The communication unit 1626 in this embodiment communicates with the outside wirelessly.

[0415] The robot 1600 is equipped with as many communication units 1626 as the communication methods used in the external devices assumed as the communication objects. Examples of the communication methods include infrared communication, visible light communication, short-range wireless communication, WiFi (registered trademark), Bluetooth (registered trademark), Radio Frequency Identification (RFID) (registered trademark), ZigBee (registered trademark), Institute of Electrical and Electronics Engineers (IEEE) 802.11a (registered trademark), MulteFire, Low Power Wide Area (LPWA), and the like.

[0416] Frequency bands used for wireless communication include short wavelength bands (eg, 800 MHz to 920 MHz), long wavelength bands (eg, 2.4 GHz, 5 GHz), and the like.

[0417] In addition, a communication cable can also be used to connect the communication unit 1626 and the external device.

[0418] The display unit 1627 may also be used to achieve visual communication with the user. For example, text or graphics may be displayed on the display unit 1627 .

[0419] When the display portion 1627 is arranged on the head 1612 , facial expressions may be displayed on the display portion 1627 .

[0420] In the present embodiment, wheels or tracks are used as the moving mechanism 1628 , but a propeller or a compressed air blowing mechanism may be used to move the robot 1600 by the force of air.

[0421] Although a secondary battery is used as the power source 1629 in this embodiment, any of a primary battery, a fuel cell, and a solar cell may be used as long as it can generate electric power.

[0422] In addition, instead of the power source 1629, a configuration in which electric power is supplied from the outside via a power cable may be adopted.

[0423] In the present embodiment, the robot 1600 is equipped with a position detection unit 1631 .

[0424] In the position detection unit 1631, for example, a method of reading location information from a Global Positioning System (GPS) signal, an Indoor Messaging System (IMES) method of locating an indoor position using a signal equivalent to GPS, a WiFi positioning method of locating a position based on the strength or arrival time of radio waves sent from multiple WiFi access points, a base station positioning method of locating a position based on the direction and delay time of a response to a signal periodically generated from a base station, a sound wave positioning method of locating a position by receiving ultrasonic waves in an inaudible area, a Bluetooth positioning method of locating a position by receiving radio waves from a beacon using Bluetooth, a visible light positioning method of locating a position using position information conveyed by flashing lighting light such as a light emitting diode (LED), an autonomous navigation method of locating the current position using an accelerometer or a gyro sensor, etc., and the like are used.

[0425] Fig.36 This is a diagram for explaining the movement of the potted plant 1500 using the robot 1600 .

[0426] exist Fig.36 In, with Fig.34 The corresponding parts are marked with corresponding symbols.

[0427] Similar to other embodiments, the user moves the right hand 310 in the space in front of the smartphone 200 and captures this movement.

[0428] Smartphone 200, which receives the movement of right hand 310 as an operation, instructs robot 1600, which can move toward potted plant 1500, to execute the operation. The operation here is to move potted plant 1500 in the lateral direction.

[0429] When receiving the instruction to move, the robot 1600 moves to the location of the potted plant 1500 to be moved, and then grabs the potted plant 1500 and moves in the lateral direction.

[0430] At this time, the image 1600A of the robot grasping the image 1500A of the potted plant and moving it and the image 310A of the right hand are displayed on the smartphone 200. Fig.36 In this case, the image 310A of the right hand is displayed as a transmitted image to make it easy to confirm the image 1600A of the robot.

[0431] exist Fig.36In this case, both the potted plant 1500 and the robot 1600 actually exist in front of the user's eyes, so there is little need to display the image 310A of the right hand through the transparent display. However, when the image 1500A of the potted plant displayed on the smartphone 200 is not in front of the user's eyes (for example, when an image of a webcam is displayed on the smartphone 200), it is easy to confirm the appearance of the real space by displaying the image 310A of the right hand through the transparent display.

[0432] <Other embodiments>

[0433] Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope described in the embodiments. As is clear from the description of the claims, various changes or improvements made to the embodiments are also included in the technical scope of the present invention.

[0434] (1) For example, in the above-described embodiment, the smartphone 200 (see Figure 1 ) or glasses-type terminals (refer to Fig.25 ) is used to operate objects existing in the real space, but as long as it has a camera for taking pictures, a display unit, and a communication unit, the information processing device that accepts the user's body movements as operations may be, for example, an input tablet terminal, a so-called digital camera, or a game console.

[0435] (2) In the above embodiment, the smartphone 200 is used to execute processing for accepting the movement of a part of one's own body photographed by the smartphone 200 as an operation, but the smartphone 200 may also be used as a component for photographing the movement of a part of the body, and an external device (such as a server) may be used to execute processing for accepting this movement as an operation.

[0436] (3) In the above embodiment, gloves are illustrated as an example of decorating a part of the user's body in accordance with the use scene, but clothes can also be added as decoration. In addition, on the screen, nails can also be painted with nail polish. In addition, when using the feet to operate, shoes corresponding to the use scene can also be added as decoration. For example, business shoes, sports shoes, slippers, boots, etc. can also be used.

[0437] (4) In the above embodiment, the case where the smartphone 200 is located on the front side of the object to be operated is described as an example of the facing position, but the facing position is not limited to the front side.

[0438] Fig.37 It is a diagram for explaining another example of a position facing the image forming apparatus 100 .

[0439] exist Fig.37, the smartphone 200 is shown as being located obliquely in front of the right, obliquely behind the right, behind, obliquely behind the left, and obliquely in front of the left, as viewed from the image forming apparatus 100. These are examples of positions facing the image forming apparatus 100.

[0440] (5) In the above-described embodiment, the image displayed on the image forming device 100 (see Figure 1 ) of the user interface unit 110 (see Figure 1 ) is described above, but the paper storage section 140 (see Figure 1 )Open and close.

[0441] Fig.38 1 is a diagram for explaining an example of opening and closing the paper storage section 140 by body movements. Since the object of the operation is the paper storage section 140, an image 140A of the storage section and an image 310A of the user's right hand are displayed on the liquid crystal display 223 of the smartphone 200. Fig.38 In the figure, as shown by the arrow, the right hand 310 is being pulled toward the front. In this case, even if the button displayed on the user interface unit 110 is not operated, the paper storage unit 140 whose overlap with the right hand image 310A is detected is pulled toward the front. Of course, as a prerequisite, a motor (not shown) for driving the storage unit 140 must be built into the image forming apparatus 100. In addition, if the right hand 310 is moved toward the back, the paper storage unit 140 can also be closed.

[0442] (6) In the above embodiment, the user interface unit 110 (see Figure 1 ), but in a case where a specific part as the object of operation is specified by body movement, the content displayed in the user interface unit 110 can also be switched to a display including a button type associated with the specified part.

[0443] Fig.39 This is a diagram for explaining an operation example in the case where a mechanism (post-processing device 150) for performing a punching process or a stapling process on printed paper is installed. In addition, the post-processing device 150 may also include a sorting mechanism.

[0444] exist Fig.39 , the image 301A of the index finger is superimposed on the image 150A of the post-processing device.

[0445] In the case of the present embodiment, the post-processing device icon 150A is assigned with a selection operation using the index finger icon 301A and a function of turning over a setting page related to the post-processing device 150 .

[0446] Fig.40 (A) to (C) in the figure are explanations of the image 150A ( Fig.39 Refer to the figure which shows the change of the screen on the display unit 111 when the selection is made by the image 301A of the index finger. Fig.40 (A) in the figure shows the display before the body movement of the index finger image 301A is received. Fig.40 (B) in the figure shows the display after receiving the body movement of the index finger image 301A. Fig.40 (C) in the figure shows an example of a screen of the smartphone 200 that receives an operation using body movements.

[0447] exist Fig.40 In (A) to (C) of FIG. 1 , the display unit 111 is photographed by the smartphone 200, and even if the operation is not performed by body movement, the screen is switched to a button 110B for specifying whether to execute the binding process. In this way, when a specific operation and a specific function are assigned to a specific part, the instruction can be completed with a small number of operations. That is, the operation efficiency can be improved.

[0448] Of course, the part to which the specific body movement and the specific function are assigned is not limited to the post-processing device 150. For example, the image forming device 100 (see Figure 1 When a selection using body movements is accepted while photographing the connection terminals or communication lines arranged on the back, a menu screen for adjusting or setting the connection terminals may be displayed on the display unit 111.

[0449] (7) In the above embodiment, the acceptance of the operation using body movements is displayed on the liquid crystal display 223 of the smartphone 200, but a function may be provided to warn the user when an erroneous operation or an unexpected operation is accepted.

[0450] Fig.41 This is a diagram illustrating an example of a function for notifying the user that a physical action of continuing to push the single-leaf door 600 after it has been opened to the maximum allowable angle has been detected.

[0451] exist Fig.41 In, with Fig.19 The corresponding parts are marked with corresponding symbols.

[0452] exist Fig.41 In this case, a small screen 290 including a warning message “Cannot open further!” is displayed at the bottom of the liquid crystal display 223.

[0453] In addition, the report can also be made by voice. In addition, the report is not limited to text and sound, and the image of the object recognized as the object of the operation can also be changed.

[0454] In addition, although the upper limit of the movable range is notified here, the detection of operation in a direction that is originally not movable may also be reported. In addition, the correct operation method may also be reported.

[0455] However, it is not necessary to report correct operations frequently. For example, when the object of the operation is a safe. The reason is that the safe does not allow unlocking by anyone other than a specific person. In addition, the method of reporting the correct operation can also be reported to an operator (authorized person, owner, etc.) who can be authenticated based on the user account of the smartphone 200 that is sending the instruction for the operation.

Claims

1. An information processing device, comprising: A camera that photographs an object that is an object of operation and exists in a real space from a position facing the object, and photographs a user's action when operating the object, wherein the user's action is performed in the real space without contact with the object; as well as a control component that determines the operation of the object corresponding to the action of the user based on the image captured by the camera, and instructs the object to perform the operation; and The information processing device is used by the user in contact with the user.

2. The information processing device according to claim 1, wherein: The camera detects the movement of the user based on the image obtained by simultaneously photographing the object and a part of the user's body.

3. The information processing device according to claim 2, wherein: The image is taken by the user.

4. The information processing device according to claim 3, wherein: A portion of the body of the user is displayed superimposed on the object.

5. The information processing device according to claim 2, wherein: The control unit associates the detected motion of the user and the object detected to overlap a part of the body of the user on the image.

6. The information processing device according to claim 5, wherein: The control component uses a preset part of the body as a detection object.

7. The information processing device according to claim 6, wherein: The preset body part is a finger or a hand.

8. The information processing device according to claim 6, wherein: The predetermined body part is a foot.

9. The information processing device according to claim 1, wherein: The camera detects the action of the user when the object and a part of the user's body are simultaneously photographed.

10. The information processing device according to claim 1, wherein: The operation corresponding to the action of the user is prepared for each individual part of the object.

11. The information processing device according to claim 1, wherein: As long as the user whose motion is detected is a user permitted to operate the object, the control unit instructs the object to perform the operation.

12. The information processing device according to claim 1, wherein: When the operation corresponding to the detected motion of the user cannot be performed in the object, a report is made to the user.

13. The information processing device according to claim 12, wherein: Report the correct method for the operation of the object.

14. The information processing device according to claim 1, wherein: The control unit determines the content of the operation corresponding to the detected action of the user based on the object.

15. The information processing device according to claim 1, wherein: The control unit acquires operable functions of the object by communicating with the object.

16. The information processing apparatus according to claim 1, wherein: The control component transparently displays the body of the user so that the object is not blocked by the body of the user.

17. The information processing device according to claim 1, wherein: The control unit selectively displays a portion of the user's body so that the object is not blocked by the user's body.

18. The information processing device according to claim 1, wherein: The control unit displays a representation suggesting a body part of the user by superimposing it on the object.

19. The information processing apparatus according to claim 1, wherein: The control unit reports to the user that the action of the user has been accepted as the operation.

20. The information processing apparatus according to claim 19, wherein: The control unit displays the body part that has been accepted as the operation in a different form from that before the acceptance.

21. The information processing apparatus according to claim 1, wherein: The control unit displays the equipment for the operation of the object by superimposing it on the body of the user.

22. The information processing apparatus according to claim 1, wherein: The control unit displays equipment corresponding to the environment in which the object exists by superimposing it on the body of the user.

23. The information processing apparatus according to claim 1, wherein: The camera detects movement of the part of the user's body based on the image obtained by simultaneously photographing a screen showing an image of the object and the part of the user's body.

24. The information processing apparatus according to claim 1, wherein: The control unit determines the content of the operation corresponding to the action of the user based on an image obtained by combining the image obtained by photographing the object and the image obtained by photographing a part of a body of the user.

25. The information processing apparatus according to claim 1, wherein: When the operation corresponding to the motion of the user requires physical operation on the object, the control unit instructs another device having a movable portion to perform the operation.

26. An information processing device, comprising: a communication component that receives an instruction to execute an operation of the information processing device from a terminal that is used by a user in contact, wherein the terminal photographs the information processing device and the actions of the user when operating the information processing device to determine the operation of the information processing device corresponding to the action of the user, wherein the action of the user is performed in a real space without contact with the information processing device; as well as A control component allows the execution of the operation instructed by the terminal.

27. The information processing apparatus according to claim 26, wherein: The control section permits execution of the operation as long as it is an instruction from the terminal.

28. A computer-readable medium storing a program, the program causing a computer to be used in contact with a user to perform information processing, the information processing comprising: Using a camera, photographing an object that is an object of operation existing in a real space from a position facing the object, and photographing a user's action when operating the object, wherein the user's action is performed in the real space without contacting the object; as well as The operation of the object corresponding to the action of the user is determined according to the image captured by the camera, and the object is instructed to perform the operation.

Images